2026-03-14 09:07:01.919221 | Job console starting...
2026-03-14 09:07:01.936944 | Updating repositories
2026-03-14 09:07:02.027977 | Preparing job workspace
2026-03-14 09:07:13.242199 | Running Ansible setup...
2026-03-14 09:07:18.318378 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/pre.yaml@master]
2026-03-14 09:07:18.892414 | 
2026-03-14 09:07:18.892532 | PLAY [localhost]
2026-03-14 09:07:18.901433 | 
2026-03-14 09:07:18.901507 | TASK [Gathering Facts]
2026-03-14 09:07:19.900146 | localhost | ok
2026-03-14 09:07:19.919985 | 
2026-03-14 09:07:19.920096 | TASK [log-inventory : Ensure Zuul Ansible directory exists]
2026-03-14 09:07:20.271083 | localhost -> localhost | changed
2026-03-14 09:07:20.285892 | 
2026-03-14 09:07:20.286133 | TASK [log-inventory : Copy ansible inventory to logs dir]
2026-03-14 09:07:21.094098 | localhost -> localhost | changed
2026-03-14 09:07:21.104589 | 
2026-03-14 09:07:21.104664 | TASK [Setup log path fact]
2026-03-14 09:07:21.122006 | localhost | ok
2026-03-14 09:07:21.133256 | 
2026-03-14 09:07:21.133325 | TASK [set-zuul-log-path-fact : Set log path for a build]
2026-03-14 09:07:21.160979 | localhost | ok
2026-03-14 09:07:21.168502 | 
2026-03-14 09:07:21.168565 | TASK [emit-job-header : Print job information]
2026-03-14 09:07:21.206455 | # Job Information
2026-03-14 09:07:21.206585 | Ansible Version: 2.15.12
2026-03-14 09:07:21.206611 | Job: watcher-operator-kuttl
2026-03-14 09:07:21.206630 | Pipeline: github-check
2026-03-14 09:07:21.206647 | Executor: ze04.softwarefactory-project.io
2026-03-14 09:07:21.206665 | Triggered by: https://github.com/openstack-k8s-operators/watcher-operator/pull/352
2026-03-14 09:07:21.206705 | Log URL (when completed): https://logserver.rdoproject.org/bd4/rdoproject.org/bd4c83535f9d4245993d8306726273fc/
2026-03-14 09:07:21.206724 | Event ID: dc9d9480-1f77-11f1-81df-d5d545c9e24e
2026-03-14 09:07:21.210450 | 
2026-03-14 09:07:21.210519 | LOOP [emit-job-header : Print node information]
2026-03-14 09:07:21.315281 | localhost | ok:
2026-03-14 09:07:21.315571 | localhost | # Node Information
2026-03-14 09:07:21.315602 | localhost | Inventory Hostname: controller
2026-03-14 09:07:21.315625 | localhost | Hostname: np0005647135
2026-03-14 09:07:21.315644 | localhost | Username: zuul
2026-03-14 09:07:21.315664 | localhost | Distro: CentOS 9
2026-03-14 09:07:21.315700 | localhost | Provider: vexxhost-nodepool-tripleo
2026-03-14 09:07:21.315719 | localhost | Region: RegionOne
2026-03-14 09:07:21.315735 | localhost | Label: cloud-centos-9-stream-tripleo-vexxhost-medium
2026-03-14 09:07:21.315751 | localhost | Product Name: OpenStack Nova
2026-03-14 09:07:21.315767 | localhost | Interface IP: 38.102.83.128
2026-03-14 09:07:21.418259 | localhost | ok:
2026-03-14 09:07:21.418426 | localhost | # Node Information
2026-03-14 09:07:21.418665 | localhost | Inventory Hostname: crc
2026-03-14 09:07:21.418719 | localhost | Hostname: crc
2026-03-14 09:07:21.418741 | localhost | Username: core
2026-03-14 09:07:21.418759 | localhost | Distro: RedHat 4.18
2026-03-14 09:07:21.418776 | localhost | Provider: vexxhost-nodepool-tripleo
2026-03-14 09:07:21.418793 | localhost | Region: RegionOne
2026-03-14 09:07:21.418810 | localhost | Label: crc-cloud-ocp-4-18-1-3xl
2026-03-14 09:07:21.418827 | localhost | Product Name: OpenStack Nova
2026-03-14 09:07:21.418843 | localhost | Interface IP: 38.102.83.162
2026-03-14 09:07:21.445330 | 
2026-03-14 09:07:21.445394 | PLAY [all]
2026-03-14 09:07:21.452115 | 
2026-03-14 09:07:21.452177 | TASK [Gather network facts]
2026-03-14 09:07:21.838537 | controller | ok
2026-03-14 09:07:22.195732 | crc | ok
2026-03-14 09:07:22.257978 | 
2026-03-14 09:07:22.258103 | TASK [include_role : start-zuul-console]
2026-03-14 09:07:22.281493 | controller | ok
2026-03-14 09:07:22.289911 | crc | ok
2026-03-14 09:07:22.307973 | 
2026-03-14 09:07:22.308086 | TASK [start-zuul-console : Start zuul_console daemon.]
2026-03-14 09:07:22.686632 | controller | ok
2026-03-14 09:07:22.725913 | crc | ok
2026-03-14 09:07:22.744091 | 
2026-03-14 09:07:22.744217 | TASK [include_role : add-build-sshkey]
2026-03-14 09:07:22.796727 | controller | ok
2026-03-14 09:07:22.803131 | crc | ok
2026-03-14 09:07:22.829348 | 
2026-03-14 09:07:22.829464 | TASK [add-build-sshkey : Check to see if ssh key was already created for this build]
2026-03-14 09:07:23.042273 | controller -> localhost | ok
2026-03-14 09:07:23.090182 | 
2026-03-14 09:07:23.090297 | TASK [add-build-sshkey : Create a new key in workspace based on build UUID]
2026-03-14 09:07:23.130981 | crc | ok
2026-03-14 09:07:23.134938 | controller | ok
2026-03-14 09:07:23.152502 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/create-key-and-replace.yaml
2026-03-14 09:07:23.152583 | controller | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/create-key-and-replace.yaml
2026-03-14 09:07:23.158906 | 
2026-03-14 09:07:23.158972 | TASK [add-build-sshkey : Create Temp SSH key]
2026-03-14 09:07:24.338870 | controller -> localhost | Generating public/private rsa key pair.
2026-03-14 09:07:24.339147 | controller -> localhost | Your identification has been saved in /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/bd4c83535f9d4245993d8306726273fc_id_rsa.
2026-03-14 09:07:24.339180 | controller -> localhost | Your public key has been saved in /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/bd4c83535f9d4245993d8306726273fc_id_rsa.pub.
2026-03-14 09:07:24.339203 | controller -> localhost | The key fingerprint is:
2026-03-14 09:07:24.339222 | controller -> localhost | SHA256:wgZ6q641enYeL4POA/N4aWQNwB1qSkg5qh+h7Ejp+H4 zuul-build-sshkey
2026-03-14 09:07:24.339241 | controller -> localhost | The key's randomart image is:
2026-03-14 09:07:24.339259 | controller -> localhost | +---[RSA 3072]----+
2026-03-14 09:07:24.339277 | controller -> localhost | |..o..            |
2026-03-14 09:07:24.339295 | controller -> localhost | |++..             |
2026-03-14 09:07:24.339312 | controller -> localhost | |o=. .            |
2026-03-14 09:07:24.339329 | controller -> localhost | |= o. o           |
2026-03-14 09:07:24.339346 | controller -> localhost | |+.o+. + S        |
2026-03-14 09:07:24.339366 | controller -> localhost | |=++..o .         |
2026-03-14 09:07:24.339391 | controller -> localhost | |*Oo+o            |
2026-03-14 09:07:24.339415 | controller -> localhost | |+*@oEo           |
2026-03-14 09:07:24.339436 | controller -> localhost | |o@B=.o.          |
2026-03-14 09:07:24.339453 | controller -> localhost | +----[SHA256]-----+
2026-03-14 09:07:24.339505 | controller -> localhost | ok: Runtime: 0:00:00.771355
2026-03-14 09:07:24.346194 | 
2026-03-14 09:07:24.346257 | TASK [add-build-sshkey : Remote setup ssh keys (linux)]
2026-03-14 09:07:24.380806 | controller | ok
2026-03-14 09:07:24.389603 | crc | ok
2026-03-14 09:07:24.407673 | controller | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/remote-linux.yaml
2026-03-14 09:07:24.407786 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/remote-linux.yaml
2026-03-14 09:07:24.420241 | 
2026-03-14 09:07:24.420358 | TASK [add-build-sshkey : Remove previously added zuul-build-sshkey]
2026-03-14 09:07:24.447175 | controller | skipping: Conditional result was False
2026-03-14 09:07:24.458027 | crc | skipping: Conditional result was False
2026-03-14 09:07:24.464005 | 
2026-03-14 09:07:24.464069 | TASK [add-build-sshkey : Enable access via build key on all nodes]
2026-03-14 09:07:24.883643 | controller | changed
2026-03-14 09:07:25.042411 | crc | changed
2026-03-14 09:07:25.047464 | 
2026-03-14 09:07:25.047532 | TASK [add-build-sshkey : Make sure user has a .ssh]
2026-03-14 09:07:25.313282 | controller | ok
2026-03-14 09:07:25.349543 | crc | ok
2026-03-14 09:07:25.360924 | 
2026-03-14 09:07:25.361065 | TASK [add-build-sshkey : Install build private key as SSH key on all nodes]
2026-03-14 09:07:26.083558 | controller | changed
2026-03-14 09:07:26.392811 | crc | changed
2026-03-14 09:07:26.404498 | 
2026-03-14 09:07:26.404671 | TASK [add-build-sshkey : Install build public key as SSH key on all nodes]
2026-03-14 09:07:27.104594 | controller | changed
2026-03-14 09:07:27.456893 | crc | changed
2026-03-14 09:07:27.468382 | 
2026-03-14 09:07:27.468525 | TASK [add-build-sshkey : Remote setup ssh keys (windows)]
2026-03-14 09:07:27.501064 | controller | skipping: Conditional result was False
2026-03-14 09:07:27.523039 | crc | skipping: Conditional result was False
2026-03-14 09:07:27.532432 | 
2026-03-14 09:07:27.532545 | TASK [remove-zuul-sshkey : Remove master key from local agent]
2026-03-14 09:07:27.907655 | controller -> localhost | changed
2026-03-14 09:07:27.930048 | 
2026-03-14 09:07:27.930194 | TASK [add-build-sshkey : Add back temp key]
2026-03-14 09:07:28.199078 | controller -> localhost | Identity added: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/bd4c83535f9d4245993d8306726273fc_id_rsa (zuul-build-sshkey)
2026-03-14 09:07:28.199295 | controller -> localhost | ok: Runtime: 0:00:00.008071
2026-03-14 09:07:28.205143 | 
2026-03-14 09:07:28.205217 | TASK [add-build-sshkey : Verify we can still SSH to all nodes]
2026-03-14 09:07:28.596258 | controller | ok
2026-03-14 09:07:28.656408 | crc | ok
2026-03-14 09:07:28.666240 | 
2026-03-14 09:07:28.666376 | TASK [add-build-sshkey : Verify we can still SSH to all nodes (windows)]
2026-03-14 09:07:28.696426 | controller | skipping: Conditional result was False
2026-03-14 09:07:28.707201 | crc | skipping: Conditional result was False
2026-03-14 09:07:28.721977 | 
2026-03-14 09:07:28.722044 | TASK [include_role : validate-host]
2026-03-14 09:07:28.742483 | controller | ok
2026-03-14 09:07:28.760164 | crc | ok
2026-03-14 09:07:28.785880 | 
2026-03-14 09:07:28.785954 | TASK [validate-host : Define zuul_info_dir fact]
2026-03-14 09:07:28.827021 | controller | ok
2026-03-14 09:07:28.832600 | crc | ok
2026-03-14 09:07:28.836994 | 
2026-03-14 09:07:28.837054 | TASK [validate-host : Ensure Zuul Ansible directory exists]
2026-03-14 09:07:29.075925 | controller -> localhost | ok
2026-03-14 09:07:29.087371 | 
2026-03-14 09:07:29.087508 | TASK [validate-host : Collect information about the host]
2026-03-14 09:07:29.690283 | controller | ok
2026-03-14 09:07:30.182061 | crc | ok
2026-03-14 09:07:30.202897 | 
2026-03-14 09:07:30.203010 | TASK [validate-host : Sanitize hostname]
2026-03-14 09:07:30.267996 | crc | ok
2026-03-14 09:07:30.282344 | controller | ok
2026-03-14 09:07:30.287102 | 
2026-03-14 09:07:30.287164 | TASK [validate-host : Write out all ansible variables/facts known for each host]
2026-03-14 09:07:30.707467 | crc -> localhost | changed
2026-03-14 09:07:30.718883 | controller -> localhost | changed
2026-03-14 09:07:30.725092 | 
2026-03-14 09:07:30.725156 | TASK [validate-host : Collect information about zuul worker]
2026-03-14 09:07:31.158770 | controller | ok
2026-03-14 09:07:31.219058 | crc | ok
2026-03-14 09:07:31.229914 | 
2026-03-14 09:07:31.230065 | TASK [validate-host : Write out all zuul information for each host]
2026-03-14 09:07:31.676608 | controller -> localhost | changed
2026-03-14 09:07:31.708083 | crc -> localhost | changed
2026-03-14 09:07:31.724359 | 
2026-03-14 09:07:31.724426 | TASK [include_role : prepare-workspace-openshift]
2026-03-14 09:07:31.748541 | controller | skipping: Conditional result was False
2026-03-14 09:07:31.753449 | 
2026-03-14 09:07:31.753508 | TASK [include_role : remove-zuul-sshkey]
2026-03-14 09:07:31.777413 | controller | skipping: Conditional result was False
2026-03-14 09:07:31.782796 | 
2026-03-14 09:07:31.782859 | LOOP [ensure-output-dirs : Empty Zuul Output directories by removing them]
2026-03-14 09:07:32.051386 | controller | ok: "logs"
2026-03-14 09:07:32.051828 | controller | ok: All items complete
2026-03-14 09:07:32.051896 | 
2026-03-14 09:07:32.099168 | crc | ok: "logs"
2026-03-14 09:07:32.280425 | controller | ok: "artifacts"
2026-03-14 09:07:32.364119 | crc | ok: "artifacts"
2026-03-14 09:07:32.480807 | controller | ok: "docs"
2026-03-14 09:07:32.628118 | crc | ok: "docs"
2026-03-14 09:07:32.643636 | 
2026-03-14 09:07:32.643794 | LOOP [ensure-output-dirs : Ensure Zuul Output directories exist]
2026-03-14 09:07:32.906348 | controller | changed: "logs"
2026-03-14 09:07:32.906588 | controller | ok: All items complete
2026-03-14 09:07:32.906618 | 
2026-03-14 09:07:32.949133 | crc | changed: "logs"
2026-03-14 09:07:33.104886 | controller | changed: "artifacts"
2026-03-14 09:07:33.229598 | crc | changed: "artifacts"
2026-03-14 09:07:33.285180 | controller | changed: "docs"
2026-03-14 09:07:33.588146 | crc | changed: "docs"
2026-03-14 09:07:33.642259 | 
2026-03-14 09:07:33.642358 | PLAY RECAP
2026-03-14 09:07:33.642400 | controller | ok: 22 changed: 9 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0
2026-03-14 09:07:33.642427 | crc | ok: 17 changed: 6 unreachable: 0 failed: 0 skipped: 3 rescued: 0 ignored: 0
2026-03-14 09:07:33.642451 | localhost | ok: 6 changed: 2 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0
2026-03-14 09:07:33.642468 | 
2026-03-14 09:07:33.760245 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/base-minimal/pre.yaml@master]
2026-03-14 09:07:33.765276 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/ci-framework-rdo-base/pre.yaml@master]
2026-03-14 09:07:34.391422 | 
2026-03-14 09:07:34.391547 | PLAY [all]
2026-03-14 09:07:34.425991 | 
2026-03-14 09:07:34.426148 | TASK [mirror-info-fork : Set mirror_fqdn if not defined]
2026-03-14 09:07:34.488702 | crc | ok
2026-03-14 09:07:34.514166 | controller | ok
2026-03-14 09:07:34.519158 | 
2026-03-14 09:07:34.519236 | TASK [mirror-info-fork : Create /etc/ci]
2026-03-14 09:07:34.932860 | controller | changed
2026-03-14 09:07:34.980642 | crc | changed
2026-03-14 09:07:34.986408 | 
2026-03-14 09:07:34.986476 | TASK [mirror-info-fork : Install ci_mirror script]
2026-03-14 09:07:35.974337 | controller | changed
2026-03-14 09:07:36.082954 | crc | changed
2026-03-14 09:07:36.112060 | 
2026-03-14 09:07:36.112195 | LOOP [add-authorized-keys : Enable access via build key on all nodes]
2026-03-14 09:07:36.519936 | controller | changed:
2026-03-14 09:07:36.520377 | controller | {
2026-03-14 09:07:36.520453 | controller |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4Z/c9osaGGtU6X8fgELwfj/yayRurfcKA0HMFfdpPxev2dbwljysMuzoVp4OZmW1gvGtyYPSNRvnzgsaabPNKNo2ym5NToCP6UM+KSe93aln4BcM/24mXChYAbXJQ5Bqq/pIzsGs/pKetQN+vwvMxLOwTvpcsCJBXaa981RKML6xj9l/UZ7IIq1HSEKMvPLxZMWdu0Ut8DkCd5F4nOw9Wgml2uYpDCj5LLCrQQ9ChdOMz8hz6SighhNlRpPkvPaet3OXxr/ytFMu7j7vv06CaEnuMMiY2aTWN1Imin9eHAylIqFHta/3gFfQSWt9jXM7owkBLKL7ATzhaAn+fjNupw== arxcruz@redhat.com"
2026-03-14 09:07:36.520504 | controller | }
2026-03-14 09:07:36.652097 | crc | changed:
2026-03-14 09:07:36.652235 | crc | {
2026-03-14 09:07:36.652297 | crc |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4Z/c9osaGGtU6X8fgELwfj/yayRurfcKA0HMFfdpPxev2dbwljysMuzoVp4OZmW1gvGtyYPSNRvnzgsaabPNKNo2ym5NToCP6UM+KSe93aln4BcM/24mXChYAbXJQ5Bqq/pIzsGs/pKetQN+vwvMxLOwTvpcsCJBXaa981RKML6xj9l/UZ7IIq1HSEKMvPLxZMWdu0Ut8DkCd5F4nOw9Wgml2uYpDCj5LLCrQQ9ChdOMz8hz6SighhNlRpPkvPaet3OXxr/ytFMu7j7vv06CaEnuMMiY2aTWN1Imin9eHAylIqFHta/3gFfQSWt9jXM7owkBLKL7ATzhaAn+fjNupw== arxcruz@redhat.com"
2026-03-14 09:07:36.652340 | crc | }
2026-03-14 09:07:36.725875 | controller | changed:
2026-03-14 09:07:36.726043 | controller | {
2026-03-14 09:07:36.726097 | controller |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDS4Fn6k4deCnIlOtLWqZJyksbepjQt04j8Ed8CGx9EKkj0fKiAxiI4TadXQYPuNHMixZy4Nevjb6aDhL5Z906TfvNHKUrjrG7G26a0k8vdc61NEQ7FmcGMWRLwwc6ReDO7lFpzYKBMk4YqfWgBuGU/K6WLKiVW2cVvwIuGIaYrE1OiiX0iVUUk7KApXlDJMXn7qjSYynfO4mF629NIp8FJal38+Kv+HA+0QkE5Y2xXnzD4Lar5+keymiCHRntPppXHeLIRzbt0gxC7v3L72hpQ3BTBEzwHpeS8KY+SX1y5lRMN45thCHfJqGmARJREDjBvWG8JXOPmVIKQtZmVcD5b mandreou@redhat.com"
2026-03-14 09:07:36.726141 | controller | }
2026-03-14 09:07:36.943613 | controller | changed:
2026-03-14 09:07:36.943797 | controller | {
2026-03-14 09:07:36.944410 | controller |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9MiLfy30deHA7xPOAlew5qUq3UP2gmRMYJi8PtkjFB20/DKeWwWNnkZPqP9AayruRoo51SIiVg870gbZE2jYl+Ncx/FYDe56JeC3ySZsXoAVkC9bP7gkOGqOmJjirvAgPMI7bogVz8i+66Q4Ar7OKTp3762G4IuWPPEg4ce4Y7lx9qWocZapHYq4cYKMxrOZ7SEbFSATBbe2bPZAPKTw8do/Eny+Hq/LkHFhIeyra6cqTFQYShr+zPln0Cr+ro/pDX3bB+1ubFgTpjpkkkQsLhDfR6cCdCWM2lgnS3BTtYj5Ct9/JRPR5YOphqZz+uB+OEu2IL68hmU9vNTth1KeX rlandy@redhat.com"
2026-03-14 09:07:36.944477 | controller | }
2026-03-14 09:07:37.027739 | crc | changed:
2026-03-14 09:07:37.027895 | crc | {
2026-03-14 09:07:37.027973 | crc |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDS4Fn6k4deCnIlOtLWqZJyksbepjQt04j8Ed8CGx9EKkj0fKiAxiI4TadXQYPuNHMixZy4Nevjb6aDhL5Z906TfvNHKUrjrG7G26a0k8vdc61NEQ7FmcGMWRLwwc6ReDO7lFpzYKBMk4YqfWgBuGU/K6WLKiVW2cVvwIuGIaYrE1OiiX0iVUUk7KApXlDJMXn7qjSYynfO4mF629NIp8FJal38+Kv+HA+0QkE5Y2xXnzD4Lar5+keymiCHRntPppXHeLIRzbt0gxC7v3L72hpQ3BTBEzwHpeS8KY+SX1y5lRMN45thCHfJqGmARJREDjBvWG8JXOPmVIKQtZmVcD5b mandreou@redhat.com"
2026-03-14 09:07:37.028019 | crc | }
2026-03-14 09:07:37.154287 | controller | changed:
2026-03-14 09:07:37.154432 | controller | {
2026-03-14 09:07:37.154484 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFCbgz8gdERiJlk2IKOtkjQxEXejrio6ZYMJAVJYpOIp raukadah@gmail.com"
2026-03-14 09:07:37.154524 | controller | }
2026-03-14 09:07:37.382173 | controller | changed:
2026-03-14 09:07:37.382342 | controller | {
2026-03-14 09:07:37.382403 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBqb3Q/9uDf4LmihQ7xeJ9gA/STIQUFPSfyyV0m8AoQi bshewale@redhat.com"
2026-03-14 09:07:37.382453 | controller | }
2026-03-14 09:07:37.431324 | crc | changed:
2026-03-14 09:07:37.431462 | crc | {
2026-03-14 09:07:37.431525 | crc |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9MiLfy30deHA7xPOAlew5qUq3UP2gmRMYJi8PtkjFB20/DKeWwWNnkZPqP9AayruRoo51SIiVg870gbZE2jYl+Ncx/FYDe56JeC3ySZsXoAVkC9bP7gkOGqOmJjirvAgPMI7bogVz8i+66Q4Ar7OKTp3762G4IuWPPEg4ce4Y7lx9qWocZapHYq4cYKMxrOZ7SEbFSATBbe2bPZAPKTw8do/Eny+Hq/LkHFhIeyra6cqTFQYShr+zPln0Cr+ro/pDX3bB+1ubFgTpjpkkkQsLhDfR6cCdCWM2lgnS3BTtYj5Ct9/JRPR5YOphqZz+uB+OEu2IL68hmU9vNTth1KeX rlandy@redhat.com"
2026-03-14 09:07:37.431593 | crc | }
2026-03-14 09:07:37.654353 | controller | changed:
2026-03-14 09:07:37.654518 | controller | {
2026-03-14 09:07:37.654570 | controller |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0I8QqQx0Az2ysJt2JuffucLijhBqnsXKEIx5GyHwxVULROa8VtNFXUDH6ZKZavhiMcmfHB2+TBTda+lDP4FldYj06dGmzCY+IYGa+uDRdxHNGYjvCfLFcmLlzRK6fNbTcui+KlUFUdKe0fb9CRoGKyhlJD5GRkM1Dv+Yb6Bj+RNnmm1fVGYxzmrD2utvffYEb0SZGWxq2R9gefx1q/3wCGjeqvufEV+AskPhVGc5T7t9eyZ4qmslkLh1/nMuaIBFcr9AUACRajsvk6mXrAN1g3HlBf2gQlhi1UEyfbqIQvzzFtsbLDlSum/KmKjy818GzvWjERfQ0VkGzCd9bSLVL dviroel@redhat.com"
2026-03-14 09:07:37.654613 | controller | }
2026-03-14 09:07:37.809629 | crc | changed:
2026-03-14 09:07:37.809826 | crc | {
2026-03-14 09:07:37.809876 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFCbgz8gdERiJlk2IKOtkjQxEXejrio6ZYMJAVJYpOIp raukadah@gmail.com"
2026-03-14 09:07:37.809935 | crc | }
2026-03-14 09:07:37.877066 | controller | changed:
2026-03-14 09:07:37.877299 | controller | {
2026-03-14 09:07:37.877362 | controller |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLOQd4ZLtkZXQGY6UwAr/06ppWQK4fDO3HaqxPk98csyOCBXsliSKK39Bso828+5srIXiW7aI6aC9P5mwi4mUZlGPfJlQbfrcGvY+b/SocuvaGK+1RrHLoJCT52LBhwgrzlXio2jeksZeein8iaTrhsPrOAs7KggIL/rB9hEiB3NaOPWhhoCP4vlW6MEMExGcqB/1FVxXFBPnLkEyW0Lk7ycVflZl2ocRxbfjZi0+tI1Wlinp8PvSQSc/WVrAcDgKjc/mB4ODPOyYy3G8FHgfMsrXSDEyjBKgLKMsdCrAUcqJQWjkqXleXSYOV4q3pzL+9umK+q/e3P/bIoSFQzmJKTU1eDfuvPXmow9F5H54fii/Da7ezlMJ+wPGHJrRAkmzvMbALy7xwswLhZMkOGNtRcPqaKYRmIBKpw3o6bCTtcNUHOtOQnzwY8JzrM2eBWJBXAANYw+9/ho80JIiwhg29CFNpVBuHbql2YxJQNrnl90guN65rYNpDxdIluweyUf8= anbanerj@kaermorhen"
2026-03-14 09:07:37.877551 | controller | }
2026-03-14 09:07:38.137389 | controller | changed:
2026-03-14 09:07:38.137551 | controller | {
2026-03-14 09:07:38.137605 | controller |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3VwV8Im9kRm49lt3tM36hj4Zv27FxGo4C1Q/0jqhzFmHY7RHbmeRr8ObhwWoHjXSozKWg8FL5ER0z3hTwL0W6lez3sL7hUaCmSuZmG5Hnl3x4vTSxDI9JZ/Y65rtYiiWQo2fC5xJhU/4+0e5e/pseCm8cKRSu+SaxhO+sd6FDojA2x1BzOzKiQRDy/1zWGp/cZkxcEuB1wHI5LMzN03c67vmbu+fhZRAUO4dQkvcnj2LrhQtpa+ytvnSjr8icMDosf1OsbSffwZFyHB/hfWGAfe0eIeSA2XPraxiPknXxiPKx2MJsaUTYbsZcm3EjFdHBBMumw5rBI74zLrMRvCO9GwBEmGT4rFng1nP+yw5DB8sn2zqpOsPg1LYRwCPOUveC13P6pgsZZPh812e8v5EKnETct+5XI3dVpdw6CnNiLwAyVAF15DJvBGT/u1k0Myg/bQn+Gv9k2MSj6LvQmf6WbZu2Wgjm30z3FyCneBqTL7mLF19YXzeC0ufHz5pnO1E= dasm@fedora"
2026-03-14 09:07:38.137668 | controller | }
2026-03-14 09:07:38.205187 | crc | changed:
2026-03-14 09:07:38.205391 | crc | {
2026-03-14 09:07:38.205467 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBqb3Q/9uDf4LmihQ7xeJ9gA/STIQUFPSfyyV0m8AoQi bshewale@redhat.com"
2026-03-14 09:07:38.205532 | crc | }
2026-03-14 09:07:38.390280 | controller | changed:
2026-03-14 09:07:38.390457 | controller | {
2026-03-14 09:07:38.390516 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHUnwjB20UKmsSed9X73eGNV5AOEFccQ3NYrRW776pEk cjeanner"
2026-03-14 09:07:38.390558 | controller | }
2026-03-14 09:07:38.593843 | crc | changed:
2026-03-14 09:07:38.593998 | crc | {
2026-03-14 09:07:38.594050 | crc |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0I8QqQx0Az2ysJt2JuffucLijhBqnsXKEIx5GyHwxVULROa8VtNFXUDH6ZKZavhiMcmfHB2+TBTda+lDP4FldYj06dGmzCY+IYGa+uDRdxHNGYjvCfLFcmLlzRK6fNbTcui+KlUFUdKe0fb9CRoGKyhlJD5GRkM1Dv+Yb6Bj+RNnmm1fVGYxzmrD2utvffYEb0SZGWxq2R9gefx1q/3wCGjeqvufEV+AskPhVGc5T7t9eyZ4qmslkLh1/nMuaIBFcr9AUACRajsvk6mXrAN1g3HlBf2gQlhi1UEyfbqIQvzzFtsbLDlSum/KmKjy818GzvWjERfQ0VkGzCd9bSLVL dviroel@redhat.com"
2026-03-14 09:07:38.594093 | crc | }
2026-03-14 09:07:38.673467 | controller | changed:
2026-03-14 09:07:38.673752 | controller | {
2026-03-14 09:07:38.673824 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDercCMGn8rW1C4P67tHgtflPdTeXlpyUJYH+6XDd2lR jgilaber@redhat.com"
2026-03-14 09:07:38.673871 | controller | }
2026-03-14 09:07:38.890888 | controller | changed:
2026-03-14 09:07:38.891032 | controller | {
2026-03-14 09:07:38.891056 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAMI6kkg9Wg0sG7jIJmyZemEBwUn1yzNpQQd3gnulOmZ adrianfuscoarnejo@gmail.com"
2026-03-14 09:07:38.891074 | controller | }
2026-03-14 09:07:38.961834 | crc | changed:
2026-03-14 09:07:38.961959 | crc | {
2026-03-14 09:07:38.962024 | crc |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLOQd4ZLtkZXQGY6UwAr/06ppWQK4fDO3HaqxPk98csyOCBXsliSKK39Bso828+5srIXiW7aI6aC9P5mwi4mUZlGPfJlQbfrcGvY+b/SocuvaGK+1RrHLoJCT52LBhwgrzlXio2jeksZeein8iaTrhsPrOAs7KggIL/rB9hEiB3NaOPWhhoCP4vlW6MEMExGcqB/1FVxXFBPnLkEyW0Lk7ycVflZl2ocRxbfjZi0+tI1Wlinp8PvSQSc/WVrAcDgKjc/mB4ODPOyYy3G8FHgfMsrXSDEyjBKgLKMsdCrAUcqJQWjkqXleXSYOV4q3pzL+9umK+q/e3P/bIoSFQzmJKTU1eDfuvPXmow9F5H54fii/Da7ezlMJ+wPGHJrRAkmzvMbALy7xwswLhZMkOGNtRcPqaKYRmIBKpw3o6bCTtcNUHOtOQnzwY8JzrM2eBWJBXAANYw+9/ho80JIiwhg29CFNpVBuHbql2YxJQNrnl90guN65rYNpDxdIluweyUf8= anbanerj@kaermorhen"
2026-03-14 09:07:38.962076 | crc | }
2026-03-14 09:07:39.117854 | controller | changed:
2026-03-14 09:07:39.118006 | controller | {
2026-03-14 09:07:39.118059 | controller |   "public_key": "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPijwpQu/3jhhhBZInXNOLEH57DrknPc3PLbsRvYyJIFzwYjX+WD4a7+nGnMYS42MuZk6TJcVqgnqofVx4isoD4= ramishra@redhat.com"
2026-03-14 09:07:39.118100 | controller | }
2026-03-14 09:07:39.331418 | crc | changed:
2026-03-14 09:07:39.331538 | crc | {
2026-03-14 09:07:39.331571 | crc |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3VwV8Im9kRm49lt3tM36hj4Zv27FxGo4C1Q/0jqhzFmHY7RHbmeRr8ObhwWoHjXSozKWg8FL5ER0z3hTwL0W6lez3sL7hUaCmSuZmG5Hnl3x4vTSxDI9JZ/Y65rtYiiWQo2fC5xJhU/4+0e5e/pseCm8cKRSu+SaxhO+sd6FDojA2x1BzOzKiQRDy/1zWGp/cZkxcEuB1wHI5LMzN03c67vmbu+fhZRAUO4dQkvcnj2LrhQtpa+ytvnSjr8icMDosf1OsbSffwZFyHB/hfWGAfe0eIeSA2XPraxiPknXxiPKx2MJsaUTYbsZcm3EjFdHBBMumw5rBI74zLrMRvCO9GwBEmGT4rFng1nP+yw5DB8sn2zqpOsPg1LYRwCPOUveC13P6pgsZZPh812e8v5EKnETct+5XI3dVpdw6CnNiLwAyVAF15DJvBGT/u1k0Myg/bQn+Gv9k2MSj6LvQmf6WbZu2Wgjm30z3FyCneBqTL7mLF19YXzeC0ufHz5pnO1E= dasm@fedora"
2026-03-14 09:07:39.331606 | crc | }
2026-03-14 09:07:39.339032 | controller | changed:
2026-03-14 09:07:39.339134 | controller | {
2026-03-14 09:07:39.339178 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICWBreHW95Wz2Toz5YwCGQwFcUG8oFYkienDh9tntmDc ralfieri@redhat.com"
2026-03-14 09:07:39.339211 | controller | }
2026-03-14 09:07:39.574940 | controller | changed:
2026-03-14 09:07:39.575070 | controller | {
2026-03-14 09:07:39.575122 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDK0iKdi8jQTpQrDdLVH/AAgLVYyTXF7AQ1gjc/5uT3t ykarel@yatinkarel"
2026-03-14 09:07:39.575166 | controller | }
2026-03-14 09:07:39.694194 | crc | changed:
2026-03-14 09:07:39.694318 | crc | {
2026-03-14 09:07:39.694349 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHUnwjB20UKmsSed9X73eGNV5AOEFccQ3NYrRW776pEk cjeanner"
2026-03-14 09:07:39.694374 | crc | }
2026-03-14 09:07:39.809158 | controller | changed:
2026-03-14 09:07:39.809277 | controller | {
2026-03-14 09:07:39.809307 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF/V/cLotA6LZeO32VL45Hd78skuA2lJA425Sm2LlQeZ fmount@horcrux"
2026-03-14 09:07:39.809332 | controller | }
2026-03-14 09:07:40.036441 | controller | changed:
2026-03-14 09:07:40.036557 | controller | {
2026-03-14 09:07:40.036595 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDa7QCjuDMVmRPo1rREbGwzYeBCYVN+Ou/3WKXZEC6Sr"
2026-03-14 09:07:40.036631 | controller | }
2026-03-14 09:07:40.064972 | crc | changed:
2026-03-14 09:07:40.065161 | crc | {
2026-03-14 09:07:40.065218 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDercCMGn8rW1C4P67tHgtflPdTeXlpyUJYH+6XDd2lR jgilaber@redhat.com"
2026-03-14 09:07:40.065252 | crc | }
2026-03-14 09:07:40.271223 | controller | changed:
2026-03-14 09:07:40.271335 | controller | {
2026-03-14 09:07:40.271365 | controller |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCfNtF7NvKl915TGsGGoseUb06Hj8L/S4toWf0hExeY+F00woL6NvBlJD0nDct+P5a22I4EhvoQCRQ8reaPCm1lybR3uiRIJsj+8zkVvLwby9LXzfZorlNG9ofjd00FEmB09uW/YvTl6Q9XwwwX6tInzIOv3TMqTHHGOL74ibbj8J/FJR0cFEyj0z4WQRvtkh32xAHl83gbuINryMt0sqRI+clj2381NKL55DRLQrVw0gsfqqxiHAnXg21qWmc4J+b9e9kiuAFQjcjwTVkwJCcg3xbPwC/qokYRby/Y5S40UUd7/jEARGXT7RZgpzTuDd1oZiCVrnrqJNPaMNdVv5MLeFdf1B7iIe5aa/fGouX7AO4SdKhZUdnJmCFAGvjC6S3JMZ2wAcUl+OHnssfmdj7XL50cLo27vjuzMtLAgSqi6N99m92WCF2s8J9aVzszX7Xz9OKZCeGsiVJp3/NdABKzSEAyM9xBD/5Vho894Sav+otpySHe3p6RUTgbB5Zu8VyZRZ/UtB3ueXxyo764yrc6qWIDqrehm84Xm9g+/jpIBzGPl07NUNJpdt/6Sgf9RIKXw/7XypO5yZfUcuFNGTxLfqjTNrtgLZNcjfav6sSdVXVcMPL//XNuRdKmVFaO76eV/oGMQGr1fGcCD+N+CpI7+Q+fCNB6VFWG4nZFuI/Iuw== averdagu@redhat.com"
2026-03-14 09:07:40.271387 | controller | }
2026-03-14 09:07:40.419807 | crc | changed:
2026-03-14 09:07:40.419912 | crc | {
2026-03-14 09:07:40.419935 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAMI6kkg9Wg0sG7jIJmyZemEBwUn1yzNpQQd3gnulOmZ adrianfuscoarnejo@gmail.com"
2026-03-14 09:07:40.419953 | crc | }
2026-03-14 09:07:40.500642 | controller | changed:
2026-03-14 09:07:40.500771 | controller | {
2026-03-14 09:07:40.500797 | controller |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq8l27xI+QlQVdS4djp9ogSoyrNE2+Ox6vKPdhSNL1J3PE5w+WCSvMz9A5gnNuH810zwbekEApbxTze/gLQJwBHA52CChfURpXrFaxY7ePXRElwKAL3mJfzBWY/c5jnNL9TCVmFJTGZkFZP3Nh+BMgZvL6xBkt3WKm6Uq18qzd9XeKcZusrA+O+uLv1fVeQnadY9RIqOCyeFYCzLWrUfTyE8x/XG0hAWIM7qpnF2cALQS2h9n4hW5ybiUN790H08wf9hFwEf5nxY9Z9dVkPFQiTSGKNBzmnCXU9skxS/xhpFjJ5duGSZdtAHe9O+nGZm9c67hxgtf8e5PDuqAdXEv2cf6e3VBAt+Bz8EKI3yosTj0oZHfwr42Yzb1l/SKy14Rggsrc9KAQlrGXan6+u2jcQqqx7l+SWmnpFiWTV9u5cWj2IgOhApOitmRBPYqk9rE2usfO0hLn/Pj/R/Nau4803e1/EikdLE7Ps95s9mX5jRDjAoUa2JwFF5RsVFyL910= ashigupt@ashigupt.remote.csb"
2026-03-14 09:07:40.500815 | controller | }
2026-03-14 09:07:40.731425 | controller | changed:
2026-03-14 09:07:40.731527 | controller | {
2026-03-14 09:07:40.731551 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOKLl0NYKwoZ/JY5KeZU8VwRAggeOxqQJeoqp3dsAaY9"
2026-03-14 09:07:40.731569 | controller | }
2026-03-14 09:07:40.806246 | crc | changed:
2026-03-14 09:07:40.806463 | crc | {
2026-03-14 09:07:40.806501 | crc |   "public_key": "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPijwpQu/3jhhhBZInXNOLEH57DrknPc3PLbsRvYyJIFzwYjX+WD4a7+nGnMYS42MuZk6TJcVqgnqofVx4isoD4= ramishra@redhat.com"
2026-03-14 09:07:40.806522 | crc | }
2026-03-14 09:07:40.958629 | controller | changed:
2026-03-14 09:07:40.958749 | controller | {
2026-03-14 09:07:40.958773 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIASASQOH2BcOyLKuuDOdWZlPi2orcjcA8q4400T73DLH evallesp@fedora"
2026-03-14 09:07:40.958856 | controller | }
2026-03-14 09:07:41.191274 | crc | changed:
2026-03-14 09:07:41.191374 | crc | {
2026-03-14 09:07:41.191397 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICWBreHW95Wz2Toz5YwCGQwFcUG8oFYkienDh9tntmDc ralfieri@redhat.com"
2026-03-14 09:07:41.191415 | crc | }
2026-03-14 09:07:41.209795 | controller | changed:
2026-03-14 09:07:41.209847 | controller | {
2026-03-14 09:07:41.209870 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILeBWlamUph+jRKV2qrx1PGU7vWuGIt5+z9k96I8WehW amsinha@amsinha-mac"
2026-03-14 09:07:41.209890 | controller | }
2026-03-14 09:07:41.446050 | controller | changed:
2026-03-14 09:07:41.446208 | controller | {
2026-03-14 09:07:41.446262 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIANvVgvJBlK3gb1yz5uef/JqIGq4HLEmY2dYA8e37swb morenod@redhat-laptop"
2026-03-14 09:07:41.446307 | controller | }
2026-03-14 09:07:41.632534 | crc | changed:
2026-03-14 09:07:41.632768 | crc | {
2026-03-14 09:07:41.632841 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDK0iKdi8jQTpQrDdLVH/AAgLVYyTXF7AQ1gjc/5uT3t ykarel@yatinkarel"
2026-03-14 09:07:41.632888 | crc | }
2026-03-14 09:07:41.659230 | controller | changed:
2026-03-14 09:07:41.659516 | controller | {
2026-03-14 09:07:41.659586 | controller |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZdI7t1cxYx65heVI24HTV4F7oQLW1zyfxHreL2TIJKxjyrUUKIFEUmTutcBlJRLNT2Eoix6x1sOw9YrchloCLcn//SGfTElr9mSc5jbjb7QXEU+zJMhtxyEJ1Po3CUGnj7ckiIXw7wcawZtrEOAQ9pH3ExYCJcEMiyNjRQZCxT3tPK+S4B95EWh5Fsrz9CkwpjNRPPH7LigCeQTM3Wc7r97utAslBUUvYceDSLA7rMgkitJE38b7rZBeYzsGQ8YYUBjTCtehqQXxCRjizbHWaaZkBU+N3zkKB6n/iCNGIO690NK7A/qb6msTijiz1PeuM8ThOsi9qXnbX5v0PoTpcFSojV7NHAQ71f0XXuS43FhZctT+Dcx44dT8Fb5vJu2cJGrk+qF8ZgJYNpRS7gPg0EG2EqjK7JMf9ULdjSu0r+KlqIAyLvtzT4eOnQipoKlb/WG5D/0ohKv7OMQ352ggfkBFIQsRXyyTCT98Ft9juqPuahi3CAQmP4H9dyE+7+Kz437PEtsxLmfm6naNmWi7Ee1DqWPwS8rEajsm4sNM4wW9gdBboJQtc0uZw0DfLj1I9r3Mc8Ol0jYtz0yNQDSzVLrGCaJlC311trU70tZ+ZkAVV6Mn8lOhSbj1cK0lvSr6ZK4dgqGl3I1eTZJJhbLNdg7UOVaiRx9543+C/p/As7w== brjackma@redhat.com"
2026-03-14 09:07:41.659632 | controller | }
2026-03-14 09:07:41.882409 | controller | changed:
2026-03-14 09:07:41.882566 | controller | {
2026-03-14 09:07:41.882614 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKwedoZ0TWPJX/z/4TAbO/kKcDZOQVgRH0hAqrL5UCI1 vcastell@redhat.com"
2026-03-14 09:07:41.882652 | controller | }
2026-03-14 09:07:42.003623 | crc | changed:
2026-03-14 09:07:42.003752 | crc | {
2026-03-14 09:07:42.003785 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF/V/cLotA6LZeO32VL45Hd78skuA2lJA425Sm2LlQeZ fmount@horcrux"
2026-03-14 09:07:42.003809 | crc | }
2026-03-14 09:07:42.331409 | controller | changed:
2026-03-14 09:07:42.331550 | controller | {
2026-03-14 09:07:42.331589 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEmv8sE8GCk6ZTPIqF0FQrttBdL3mq7rCm/IJy0xDFh7 michburk@redhat.com"
2026-03-14 09:07:42.331618 | controller | }
2026-03-14 09:07:42.458196 | crc | changed:
2026-03-14 09:07:42.458313 | crc | {
2026-03-14 09:07:42.458344 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDa7QCjuDMVmRPo1rREbGwzYeBCYVN+Ou/3WKXZEC6Sr"
2026-03-14 09:07:42.458370 | crc | }
2026-03-14 09:07:42.588079 | controller | changed:
2026-03-14 09:07:42.588248 | controller | {
2026-03-14 09:07:42.588302 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICy6GpGEtwevXEEn4mmLR5lmSLe23dGgAvzkB9DMNbkf rsafrono@rsafrono"
2026-03-14 09:07:42.588345 | controller | }
2026-03-14 09:07:42.788098 | crc | changed:
2026-03-14 09:07:42.788414 | crc | {
2026-03-14 09:07:42.788481 | crc |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCfNtF7NvKl915TGsGGoseUb06Hj8L/S4toWf0hExeY+F00woL6NvBlJD0nDct+P5a22I4EhvoQCRQ8reaPCm1lybR3uiRIJsj+8zkVvLwby9LXzfZorlNG9ofjd00FEmB09uW/YvTl6Q9XwwwX6tInzIOv3TMqTHHGOL74ibbj8J/FJR0cFEyj0z4WQRvtkh32xAHl83gbuINryMt0sqRI+clj2381NKL55DRLQrVw0gsfqqxiHAnXg21qWmc4J+b9e9kiuAFQjcjwTVkwJCcg3xbPwC/qokYRby/Y5S40UUd7/jEARGXT7RZgpzTuDd1oZiCVrnrqJNPaMNdVv5MLeFdf1B7iIe5aa/fGouX7AO4SdKhZUdnJmCFAGvjC6S3JMZ2wAcUl+OHnssfmdj7XL50cLo27vjuzMtLAgSqi6N99m92WCF2s8J9aVzszX7Xz9OKZCeGsiVJp3/NdABKzSEAyM9xBD/5Vho894Sav+otpySHe3p6RUTgbB5Zu8VyZRZ/UtB3ueXxyo764yrc6qWIDqrehm84Xm9g+/jpIBzGPl07NUNJpdt/6Sgf9RIKXw/7XypO5yZfUcuFNGTxLfqjTNrtgLZNcjfav6sSdVXVcMPL//XNuRdKmVFaO76eV/oGMQGr1fGcCD+N+CpI7+Q+fCNB6VFWG4nZFuI/Iuw== averdagu@redhat.com"
2026-03-14 09:07:42.788527 | crc | }
2026-03-14 09:07:42.788602 | crc | changed: All items complete
2026-03-14 09:07:42.788650 | 
2026-03-14 09:07:43.182919 | crc | changed:
2026-03-14 09:07:43.183078 | crc | {
2026-03-14 09:07:43.183129 | crc |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq8l27xI+QlQVdS4djp9ogSoyrNE2+Ox6vKPdhSNL1J3PE5w+WCSvMz9A5gnNuH810zwbekEApbxTze/gLQJwBHA52CChfURpXrFaxY7ePXRElwKAL3mJfzBWY/c5jnNL9TCVmFJTGZkFZP3Nh+BMgZvL6xBkt3WKm6Uq18qzd9XeKcZusrA+O+uLv1fVeQnadY9RIqOCyeFYCzLWrUfTyE8x/XG0hAWIM7qpnF2cALQS2h9n4hW5ybiUN790H08wf9hFwEf5nxY9Z9dVkPFQiTSGKNBzmnCXU9skxS/xhpFjJ5duGSZdtAHe9O+nGZm9c67hxgtf8e5PDuqAdXEv2cf6e3VBAt+Bz8EKI3yosTj0oZHfwr42Yzb1l/SKy14Rggsrc9KAQlrGXan6+u2jcQqqx7l+SWmnpFiWTV9u5cWj2IgOhApOitmRBPYqk9rE2usfO0hLn/Pj/R/Nau4803e1/EikdLE7Ps95s9mX5jRDjAoUa2JwFF5RsVFyL910= ashigupt@ashigupt.remote.csb"
2026-03-14 09:07:43.183171 | crc | }
2026-03-14 09:07:43.545290 | crc | changed:
2026-03-14 09:07:43.545480 | crc | {
2026-03-14 09:07:43.545552 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOKLl0NYKwoZ/JY5KeZU8VwRAggeOxqQJeoqp3dsAaY9"
2026-03-14 09:07:43.545607 | crc | }
2026-03-14 09:07:43.934787 | crc | changed:
2026-03-14 09:07:43.934952 | crc | {
2026-03-14 09:07:43.935002 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIASASQOH2BcOyLKuuDOdWZlPi2orcjcA8q4400T73DLH evallesp@fedora"
2026-03-14 09:07:43.935041 | crc | }
2026-03-14 09:07:44.316381 | crc | changed:
2026-03-14 09:07:44.316545 | crc | {
2026-03-14 09:07:44.316613 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILeBWlamUph+jRKV2qrx1PGU7vWuGIt5+z9k96I8WehW amsinha@amsinha-mac"
2026-03-14 09:07:44.316672 | crc | }
2026-03-14 09:07:44.653252 | crc | changed:
2026-03-14 09:07:44.653356 | crc | {
2026-03-14 09:07:44.653380 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIANvVgvJBlK3gb1yz5uef/JqIGq4HLEmY2dYA8e37swb morenod@redhat-laptop"
2026-03-14 09:07:44.653398 | crc | }
2026-03-14 09:07:45.020850 | crc | changed:
2026-03-14 09:07:45.021159 | crc | {
2026-03-14 09:07:45.021227 | crc |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZdI7t1cxYx65heVI24HTV4F7oQLW1zyfxHreL2TIJKxjyrUUKIFEUmTutcBlJRLNT2Eoix6x1sOw9YrchloCLcn//SGfTElr9mSc5jbjb7QXEU+zJMhtxyEJ1Po3CUGnj7ckiIXw7wcawZtrEOAQ9pH3ExYCJcEMiyNjRQZCxT3tPK+S4B95EWh5Fsrz9CkwpjNRPPH7LigCeQTM3Wc7r97utAslBUUvYceDSLA7rMgkitJE38b7rZBeYzsGQ8YYUBjTCtehqQXxCRjizbHWaaZkBU+N3zkKB6n/iCNGIO690NK7A/qb6msTijiz1PeuM8ThOsi9qXnbX5v0PoTpcFSojV7NHAQ71f0XXuS43FhZctT+Dcx44dT8Fb5vJu2cJGrk+qF8ZgJYNpRS7gPg0EG2EqjK7JMf9ULdjSu0r+KlqIAyLvtzT4eOnQipoKlb/WG5D/0ohKv7OMQ352ggfkBFIQsRXyyTCT98Ft9juqPuahi3CAQmP4H9dyE+7+Kz437PEtsxLmfm6naNmWi7Ee1DqWPwS8rEajsm4sNM4wW9gdBboJQtc0uZw0DfLj1I9r3Mc8Ol0jYtz0yNQDSzVLrGCaJlC311trU70tZ+ZkAVV6Mn8lOhSbj1cK0lvSr6ZK4dgqGl3I1eTZJJhbLNdg7UOVaiRx9543+C/p/As7w== brjackma@redhat.com"
2026-03-14 09:07:45.021270 | crc | }
2026-03-14 09:07:45.430562 | crc | changed:
2026-03-14 09:07:45.430768 | crc | {
2026-03-14 09:07:45.430833 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKwedoZ0TWPJX/z/4TAbO/kKcDZOQVgRH0hAqrL5UCI1 vcastell@redhat.com"
2026-03-14 09:07:45.430893 | crc | }
2026-03-14 09:07:45.842405 | crc | changed:
2026-03-14 09:07:45.842777 | crc | {
2026-03-14 09:07:45.842848 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEmv8sE8GCk6ZTPIqF0FQrttBdL3mq7rCm/IJy0xDFh7 michburk@redhat.com"
2026-03-14 09:07:45.842910 | crc | }
2026-03-14 09:07:46.219630 | crc | changed:
2026-03-14 09:07:46.219787 | crc | {
2026-03-14 09:07:46.219839 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICy6GpGEtwevXEEn4mmLR5lmSLe23dGgAvzkB9DMNbkf rsafrono@rsafrono"
2026-03-14 09:07:46.219877 | crc | }
2026-03-14 09:07:46.276092 | 
2026-03-14 09:07:46.276238 | TASK [Set timezone to UTC]
2026-03-14 09:07:46.829939 | controller | changed: executed `/bin/timedatectl set-timezone UTC`
2026-03-14 09:07:47.008083 | crc | ok
2026-03-14 09:07:47.019273 | 
2026-03-14 09:07:47.019412 | TASK [Create nodepool directory]
2026-03-14 09:07:47.285919 | controller | changed
2026-03-14 09:07:47.393114 | crc | changed
2026-03-14 09:07:47.405749 | 
2026-03-14 09:07:47.405888 | TASK [Create nodepool sub_nodes file]
2026-03-14 09:07:48.025245 | controller | changed
2026-03-14 09:07:48.257944 | crc | changed
2026-03-14 09:07:48.269819 | 
2026-03-14 09:07:48.270017 | TASK [Create nodepool sub_nodes_private file]
2026-03-14 09:07:48.870131 | controller | changed
2026-03-14 09:07:49.222649 | crc | changed
2026-03-14 09:07:49.235260 | 
2026-03-14 09:07:49.235424 | LOOP [Populate nodepool sub_nodes file]
2026-03-14 09:07:49.311350 | 
2026-03-14 09:07:49.311552 | LOOP [Populate nodepool sub_nodes_private file]
2026-03-14 09:07:49.372300 | 
2026-03-14 09:07:49.372494 | TASK [Create nodepool primary file]
2026-03-14 09:07:49.413494 | controller | skipping: Conditional result was False
2026-03-14 09:07:49.424425 | crc | skipping: Conditional result was False
2026-03-14 09:07:49.431771 | 
2026-03-14 09:07:49.431855 | TASK [Create nodepool node_private for this node]
2026-03-14 09:07:50.140604 | controller | changed
2026-03-14 09:07:50.381630 | crc | changed
2026-03-14 09:07:50.394163 | 
2026-03-14 09:07:50.394296 | LOOP [Copy ssh keys to nodepool directory]
2026-03-14 09:07:50.761752 | controller | ok: Item: id_rsa Runtime: 0:00:00.003192
2026-03-14 09:07:50.762138 | 
2026-03-14 09:07:50.860038 | crc | ok: Item: id_rsa Runtime: 0:00:00.008361
2026-03-14 09:07:50.933802 | controller | ok: Item: id_rsa.pub Runtime: 0:00:00.003585
2026-03-14 09:07:51.129783 | crc | ok: Item: id_rsa.pub Runtime: 0:00:00.006480
2026-03-14 09:07:51.146967 | 
2026-03-14 09:07:51.147110 | TASK [Add sudoers role for zuul-sudo-grep.sh]
2026-03-14 09:07:51.791701 | controller | changed
2026-03-14 09:07:52.106254 | crc | changed
2026-03-14 09:07:52.118317 | 
2026-03-14 09:07:52.118448 | TASK [Validate sudoers config after edits]
2026-03-14 09:07:52.455455 | controller | /etc/sudoers: parsed OK
2026-03-14 09:07:52.455576 | controller | /etc/sudoers.d/90-cloud-init-users: parsed OK
2026-03-14 09:07:52.455584 | controller | /etc/sudoers.d/zuul: parsed OK
2026-03-14 09:07:52.455671 | controller | /etc/sudoers.d/zuul-sudo-grep: parsed OK
2026-03-14 09:07:52.526215 | crc | /etc/sudoers: parsed OK
2026-03-14 09:07:52.526425 | crc | /etc/sudoers.d/coreos-sudo-group: parsed OK
2026-03-14 09:07:52.526469 | crc | /etc/sudoers.d/zuul-sudo-grep: parsed OK
2026-03-14 09:07:52.705773 | controller | ok: Runtime: 0:00:00.008778
2026-03-14 09:07:52.720253 | crc | ok: Runtime: 0:00:00.012197
2026-03-14 09:07:52.731513 | 
2026-03-14 09:07:52.731641 | TASK [Show the environment passed in to job shell scripts]
2026-03-14 09:07:53.041007 | controller | SHELL=/bin/bash
2026-03-14 09:07:53.041054 | controller | ZUUL_PROJECT=openstack-k8s-operators/watcher-operator
2026-03-14 09:07:53.041061 | controller | ZUUL_SHORT_PROJECT_NAME=watcher-operator
2026-03-14 09:07:53.041067 | controller | ZUUL_CHANGES=openstack-k8s-operators/watcher-operator:main:refs/changes/52/352/0d9467a022a6d6a8b6465cdbc9e4aefc533e796a
2026-03-14 09:07:53.041073 | controller | PWD=/home/zuul
2026-03-14 09:07:53.041078 | controller | ZUUL_PIPELINE=github-check
2026-03-14 09:07:53.041083 | controller | LOGNAME=zuul
2026-03-14 09:07:53.041087 | controller | XDG_SESSION_TYPE=tty
2026-03-14 09:07:53.041092 | controller | _=/usr/bin/env
2026-03-14 09:07:53.041096 | controller | MOTD_SHOWN=pam
2026-03-14 09:07:53.041101 | controller | HOME=/home/zuul
2026-03-14 09:07:53.041105 | controller | LANG=en_US.UTF-8
2026-03-14 09:07:53.041110 | controller | SSH_CONNECTION=38.102.83.114 44714 38.102.83.128 22
2026-03-14 09:07:53.041115 | controller | TOX_TESTENV_PASSENV=ZUUL_UUID ZUUL_PROJECT ZUUL_SHORT_PROJECT_NAME ZUUL_PIPELINE ZUUL_VOTING WORKSPACE BUILD_TIMEOUT ZUUL_BRANCH ZUUL_CHANGES ZUUL_CHANGE_IDS ZUUL_CHANGE ZUUL_PATCHSET
2026-03-14 09:07:53.041122 | controller | ZUUL_CHANGE_IDS=352,0d9467a022a6d6a8b6465cdbc9e4aefc533e796a
2026-03-14 09:07:53.041127 | controller | WORKSPACE=/home/zuul/workspace
2026-03-14 09:07:53.041132 | controller | XDG_SESSION_CLASS=user
2026-03-14 09:07:53.041136 | controller | SELINUX_ROLE_REQUESTED=
2026-03-14 09:07:53.041141 | controller | LESSOPEN=||/usr/bin/lesspipe.sh %s
2026-03-14 09:07:53.041146 | controller | USER=zuul
2026-03-14 09:07:53.041151 | controller | ZUUL_VOTING=True
2026-03-14 09:07:53.041155 | controller | BUILD_TIMEOUT=7200000
2026-03-14 09:07:53.041160 | controller | SELINUX_USE_CURRENT_RANGE=
2026-03-14 09:07:53.041164 | controller | SHLVL=1
2026-03-14 09:07:53.041169 | controller | ZUUL_PATCHSET=0d9467a022a6d6a8b6465cdbc9e4aefc533e796a
2026-03-14 09:07:53.041174 | controller | XDG_SESSION_ID=1
2026-03-14 09:07:53.041178 | controller | ZUUL_BRANCH=main
2026-03-14 09:07:53.041183 | controller | XDG_RUNTIME_DIR=/run/user/1000
2026-03-14 09:07:53.041187 | controller | SSH_CLIENT=38.102.83.114 44714 22
2026-03-14 09:07:53.041192 | controller | DEBUGINFOD_URLS=https://debuginfod.centos.org/
2026-03-14 09:07:53.041197 | controller | DEBUGINFOD_IMA_CERT_PATH=/etc/keys/ima:
2026-03-14 09:07:53.041202 | controller | which_declare=declare -f
2026-03-14 09:07:53.041208 | controller | PATH=/home/zuul/.local/bin:/home/zuul/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
2026-03-14 09:07:53.041212 | controller | SELINUX_LEVEL_REQUESTED=
2026-03-14 09:07:53.041217 | controller | ZUUL_CHANGE=352
2026-03-14 09:07:53.041222 | controller | DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
2026-03-14 09:07:53.041226 | controller | ZUUL_UUID=bd4c83535f9d4245993d8306726273fc
2026-03-14 09:07:53.041231 | controller | BASH_FUNC_which%%=() {  ( alias;
2026-03-14 09:07:53.041235 | controller |  eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@
2026-03-14 09:07:53.041240 | controller | }
2026-03-14 09:07:53.108293 | crc | SHELL=/bin/bash
2026-03-14 09:07:53.108372 | crc | ZUUL_PROJECT=openstack-k8s-operators/watcher-operator
2026-03-14 09:07:53.108390 | crc | ZUUL_SHORT_PROJECT_NAME=watcher-operator
2026-03-14 09:07:53.108404 | crc | ZUUL_CHANGES=openstack-k8s-operators/watcher-operator:main:refs/changes/52/352/0d9467a022a6d6a8b6465cdbc9e4aefc533e796a
2026-03-14 09:07:53.108417 | crc | PWD=/var/home/core
2026-03-14 09:07:53.108430 | crc | ZUUL_PIPELINE=github-check
2026-03-14 09:07:53.108471 | crc | LOGNAME=core
2026-03-14 09:07:53.108484 | crc | XDG_SESSION_TYPE=tty
2026-03-14 09:07:53.108496 | crc | _=/usr/bin/env
2026-03-14 09:07:53.108517 | crc | MOTD_SHOWN=pam
2026-03-14 09:07:53.108541 | crc | HOME=/var/home/core
2026-03-14 09:07:53.108638 | crc | LANG=C.UTF-8
2026-03-14 09:07:53.108656 | crc | SSH_CONNECTION=38.102.83.114 44144 38.102.83.162 22
2026-03-14 09:07:53.108670 | crc | TOX_TESTENV_PASSENV=ZUUL_UUID ZUUL_PROJECT ZUUL_SHORT_PROJECT_NAME ZUUL_PIPELINE ZUUL_VOTING WORKSPACE BUILD_TIMEOUT ZUUL_BRANCH ZUUL_CHANGES ZUUL_CHANGE_IDS ZUUL_CHANGE ZUUL_PATCHSET
2026-03-14 09:07:53.108688 | crc | ZUUL_CHANGE_IDS=352,0d9467a022a6d6a8b6465cdbc9e4aefc533e796a
2026-03-14 09:07:53.108701 | crc | WORKSPACE=/home/zuul/workspace
2026-03-14 09:07:53.108714 | crc | XDG_SESSION_CLASS=user
2026-03-14 09:07:53.108726 | crc | SELINUX_ROLE_REQUESTED=
2026-03-14 09:07:53.108739 | crc | LESSOPEN=||/usr/bin/lesspipe.sh %s
2026-03-14 09:07:53.108751 | crc | USER=core
2026-03-14 09:07:53.108764 | crc | ZUUL_VOTING=True
2026-03-14 09:07:53.108776 | crc | BUILD_TIMEOUT=7200000
2026-03-14 09:07:53.108788 | crc | SELINUX_USE_CURRENT_RANGE=
2026-03-14 09:07:53.108801 | crc | SHLVL=1
2026-03-14 09:07:53.108813 | crc | ZUUL_PATCHSET=0d9467a022a6d6a8b6465cdbc9e4aefc533e796a
2026-03-14 09:07:53.108826 | crc | XDG_SESSION_ID=2
2026-03-14 09:07:53.108838 | crc | ZUUL_BRANCH=main
2026-03-14 09:07:53.108850 | crc | XDG_RUNTIME_DIR=/run/user/1000
2026-03-14 09:07:53.108863 | crc | SSH_CLIENT=38.102.83.114 44144 22
2026-03-14 09:07:53.108878 | crc | which_declare=declare -f
2026-03-14 09:07:53.108890 | crc | PATH=/var/home/core/.local/bin:/var/home/core/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
2026-03-14 09:07:53.108903 | crc | SELINUX_LEVEL_REQUESTED=
2026-03-14 09:07:53.108916 | crc | ZUUL_CHANGE=352
2026-03-14 09:07:53.108928 | crc | DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
2026-03-14 09:07:53.108941 | crc | ZUUL_UUID=bd4c83535f9d4245993d8306726273fc
2026-03-14 09:07:53.108953 | crc | BASH_FUNC_which%%=() {  ( alias;
2026-03-14 09:07:53.108966 | crc |  eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@
2026-03-14 09:07:53.108979 | crc | }
2026-03-14 09:07:53.324888 | controller | ok: Runtime: 0:00:00.006571
2026-03-14 09:07:53.340322 | crc | ok: Runtime: 0:00:00.012100
2026-03-14 09:07:53.351658 | 
2026-03-14 09:07:53.351840 | TASK [Workaround hardcoded /home/zuul/workspace in zuul_legacy_vars]
2026-03-14 09:07:53.383039 | controller | skipping: Conditional result was False
2026-03-14 09:07:53.404438 | crc | skipping: Conditional result was False
2026-03-14 09:07:53.413402 | 
2026-03-14 09:07:53.413491 | TASK [Symlink /home/zuul-worker/workspace]
2026-03-14 09:07:53.990148 | controller | skipping: Conditional result was False
2026-03-14 09:07:53.991295 | crc | skipping: Conditional result was False
2026-03-14 09:07:54.005830 | 
2026-03-14 09:07:54.006023 | TASK [Ensure legacy workspace directory]
2026-03-14 09:07:54.284788 | controller | changed
2026-03-14 09:07:54.341876 | crc | changed
2026-03-14 09:07:54.391409 | 
2026-03-14 09:07:54.391509 | PLAY RECAP
2026-03-14 09:07:54.391551 | controller | ok: 14 changed: 11 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0
2026-03-14 09:07:54.391576 | crc | ok: 14 changed: 10 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0
2026-03-14 09:07:54.391594 | 
2026-03-14 09:07:54.512284 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/ci-framework-rdo-base/pre.yaml@master]
2026-03-14 09:07:54.516530 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network.yml@master]
2026-03-14 09:07:55.131038 | 
2026-03-14 09:07:55.131145 | PLAY [controller]
2026-03-14 09:07:55.153147 | 
2026-03-14 09:07:55.153235 | TASK [Debug parameters]
2026-03-14 09:07:55.191430 | controller | ok
2026-03-14 09:07:55.196732 | 
2026-03-14 09:07:55.196801 | TASK [Assert crc_ci_bootstrap_cloud_name is defined]
2026-03-14 09:07:55.237066 | controller | ok: All assertions passed
2026-03-14 09:07:55.243447 | 
2026-03-14 09:07:55.243532 | TASK [Assert crc_ci_bootstrap_networking is defined and contains the minimum fields]
2026-03-14 09:07:55.284281 | controller | ok: All assertions passed
2026-03-14 09:07:55.290697 | 
2026-03-14 09:07:55.290786 | TASK [Assert that the default network exists and has no VLAN tag]
2026-03-14 09:07:55.324128 | controller | ok: All assertions passed
2026-03-14 09:07:55.330999 | 
2026-03-14 09:07:55.331086 | LOOP [Assert that all instances has a given fixed IP for each net and default net is present]
2026-03-14 09:07:55.450380 | controller | ok: All assertions passed
2026-03-14 09:07:55.463064 | controller | ok: All assertions passed
2026-03-14 09:07:55.473414 | 
2026-03-14 09:07:55.473502 | LOOP [Assert that every host references an valid/declared network]
2026-03-14 09:07:55.534374 | controller | ok: All assertions passed
2026-03-14 09:07:55.534641 | controller | ok: All items complete
2026-03-14 09:07:55.534697 | 
2026-03-14 09:07:55.544401 | controller | ok: All assertions passed
2026-03-14 09:07:55.557284 | 
2026-03-14 09:07:55.557368 | LOOP [Assert that all instances has in range fixed IP for each net]
2026-03-14 09:07:55.643500 | controller | ok: All assertions passed
2026-03-14 09:07:55.664504 | controller | ok: All assertions passed
2026-03-14 09:07:55.677424 | 
2026-03-14 09:07:55.677566 | TASK [Assert that all non-default nets has the vlan tag]
2026-03-14 09:07:55.751822 | controller | ok: All assertions passed
2026-03-14 09:07:55.758329 | 
2026-03-14 09:07:55.758410 | TASK [Install required packages]
2026-03-14 09:08:10.675947 | controller | changed
2026-03-14 09:08:10.681791 | 
2026-03-14 09:08:10.681849 | TASK [Create openstack config dir]
2026-03-14 09:08:11.079389 | controller | changed
2026-03-14 09:08:11.091888 | 
2026-03-14 09:08:11.092019 | LOOP [Create ci dir in etc folder in the controller]
2026-03-14 09:08:11.467336 | controller -> 38.102.83.162 | changed: "crc"
2026-03-14 09:08:11.467872 | controller -> 38.102.83.162 | ok: All items complete
2026-03-14 09:08:11.467946 | 
2026-03-14 09:08:11.683915 | controller -> 38.102.83.128 | changed: "controller"
2026-03-14 09:08:11.708606 | 
2026-03-14 09:08:11.708791 | TASK [Set global resource names as facts]
2026-03-14 09:08:11.772882 | controller | ok
2026-03-14 09:08:11.779874 | 
2026-03-14 09:08:11.779942 | TASK [Generate CA cert file]
2026-03-14 09:08:11.813817 | controller | skipping: Conditional result was False
2026-03-14 09:08:11.820914 | 
2026-03-14 09:08:11.820993 | TASK [Update CA certs]
2026-03-14 09:08:12.353025 | controller | skipping: Conditional result was False
2026-03-14 09:08:12.370494 | 
2026-03-14 09:08:12.370673 | TASK [Generate clouds config from cloud_secrets secret]
2026-03-14 09:08:13.341767 | controller | changed
2026-03-14 09:08:13.346888 | 
2026-03-14 09:08:13.346951 | TASK [Create the default network]
2026-03-14 09:08:17.620749 | controller | admin_state_up: true
2026-03-14 09:08:17.620884 | controller | availability_zone_hints:
2026-03-14 09:08:17.621020 | controller | - nova
2026-03-14 09:08:17.621038 | controller | availability_zones: []
2026-03-14 09:08:17.621250 | controller | created_at: '2026-03-14T09:08:15Z'
2026-03-14 09:08:17.621362 | controller | description: ''
2026-03-14 09:08:17.621446 | controller | dns_domain: ''
2026-03-14 09:08:17.621561 | controller | id: 993bd7ea-a6c0-4049-a480-d39fe80f9b55
2026-03-14 09:08:17.621707 | controller | ipv4_address_scope: null
2026-03-14 09:08:17.621803 | controller | ipv6_address_scope: null
2026-03-14 09:08:17.621923 | controller | is_default: false
2026-03-14 09:08:17.622015 | controller | is_vlan_qinq: null
2026-03-14 09:08:17.622110 | controller | is_vlan_transparent: true
2026-03-14 09:08:17.622201 | controller | l2_adjacency: true
2026-03-14 09:08:17.622333 | controller | mtu: 1500
2026-03-14 09:08:17.622437 | controller | name: zuul-ci-net-bd4c8353
2026-03-14 09:08:17.622536 | controller | port_security_enabled: false
2026-03-14 09:08:17.622679 | controller | project_id: 4b633c451ac74233be3721a3635275e5
2026-03-14 09:08:17.622779 | controller | provider:network_type: null
2026-03-14 09:08:17.622879 | controller | provider:physical_network: null
2026-03-14 09:08:17.622978 | controller | provider:segmentation_id: null
2026-03-14 09:08:17.623069 | controller | qos_policy_id: null
2026-03-14 09:08:17.623160 | controller | revision_number: 1
2026-03-14 09:08:17.623266 | controller | router:external: false
2026-03-14 09:08:17.623364 | controller | segments: null
2026-03-14 09:08:17.623455 | controller | shared: false
2026-03-14 09:08:17.623540 | controller | status: ACTIVE
2026-03-14 09:08:17.623640 | controller | subnets: []
2026-03-14 09:08:17.623727 | controller | tags: []
2026-03-14 09:08:17.623837 | controller | updated_at: '2026-03-14T09:08:15Z'
2026-03-14 09:08:17.881375 | controller | changed
2026-03-14 09:08:17.890604 | 
2026-03-14 09:08:17.890763 | TASK [Create subnet - ipv4]
2026-03-14 09:08:21.039351 | controller | allocation_pools:
2026-03-14 09:08:21.039607 | controller | - end: 192.168.122.254
2026-03-14 09:08:21.039621 | controller |   start: 192.168.122.2
2026-03-14 09:08:21.039630 | controller | cidr: 192.168.122.0/24
2026-03-14 09:08:21.039772 | controller | created_at: '2026-03-14T09:08:20Z'
2026-03-14 09:08:21.039891 | controller | description: ''
2026-03-14 09:08:21.040010 | controller | dns_nameservers: []
2026-03-14 09:08:21.040163 | controller | dns_publish_fixed_ip: null
2026-03-14 09:08:21.040309 | controller | enable_dhcp: false
2026-03-14 09:08:21.040435 | controller | gateway_ip: 192.168.122.1
2026-03-14 09:08:21.040549 | controller | host_routes: []
2026-03-14 09:08:21.040707 | controller | id: 3293acd3-87a9-4c81-8422-4b391ba88681
2026-03-14 09:08:21.040852 | controller | ip_version: 4
2026-03-14 09:08:21.040972 | controller | ipv6_address_mode: null
2026-03-14 09:08:21.041114 | controller | ipv6_ra_mode: null
2026-03-14 09:08:21.041241 | controller | name: zuul-ci-subnet-bd4c8353
2026-03-14 09:08:21.041380 | controller | network_id: 993bd7ea-a6c0-4049-a480-d39fe80f9b55
2026-03-14 09:08:21.041515 | controller | project_id: 4b633c451ac74233be3721a3635275e5
2026-03-14 09:08:21.041647 | controller | revision_number: 0
2026-03-14 09:08:21.041764 | controller | segment_id: null
2026-03-14 09:08:21.041877 | controller | service_types: []
2026-03-14 09:08:21.041991 | controller | subnetpool_id: null
2026-03-14 09:08:21.042096 | controller | tags: []
2026-03-14 09:08:21.042227 | controller | updated_at: '2026-03-14T09:08:20Z'
2026-03-14 09:08:21.446536 | controller | changed
2026-03-14 09:08:21.459209 | 
2026-03-14 09:08:21.459398 | TASK [Set yaml returned data for further usage]
2026-03-14 09:08:21.553502 | controller | ok
2026-03-14 09:08:21.566235 | 
2026-03-14 09:08:21.566373 | TASK [Create router]
2026-03-14 09:08:22.116254 | controller | skipping: Conditional result was False
2026-03-14 09:08:22.130182 | 
2026-03-14 09:08:22.130476 | TASK [Set router yaml returned data for further usage]
2026-03-14 09:08:22.198345 | controller | skipping: Conditional result was False
2026-03-14 09:08:22.211109 | 
2026-03-14 09:08:22.211502 | TASK [Add router to subnet]
2026-03-14 09:08:22.764098 | controller | skipping: Conditional result was False
2026-03-14 09:08:22.777614 | 
2026-03-14 09:08:22.777809 | LOOP [Create ports, VLANs and configure them on the instance]
2026-03-14 09:08:22.855774 | controller | ok:
2026-03-14 09:08:22.856541 | controller | {
2026-03-14 09:08:22.856613 | controller |   "key": "controller",
2026-03-14 09:08:22.856657 | controller |   "value": {
2026-03-14 09:08:22.856733 | controller |     "networks": {
2026-03-14 09:08:22.856773 | controller |       "default": {
2026-03-14 09:08:22.856809 | controller |         "ip": "192.168.122.11"
2026-03-14 09:08:22.856844 | controller |       }
2026-03-14 09:08:22.856878 | controller |     }
2026-03-14 09:08:22.856912 | controller |   }
2026-03-14 09:08:22.856945 | controller | }
2026-03-14 09:08:25.849699 | controller | ok:
2026-03-14 09:08:25.849810 | controller | {
2026-03-14 09:08:25.849838 | controller |   "key": "crc",
2026-03-14 09:08:25.849858 | controller |   "value": {
2026-03-14 09:08:25.849875 | controller |     "networks": {
2026-03-14 09:08:25.849891 | controller |       "default": {
2026-03-14 09:08:25.849907 | controller |         "ip": "192.168.122.10"
2026-03-14 09:08:25.849926 | controller |       },
2026-03-14 09:08:25.849951 | controller |       "internal-api": {
2026-03-14 09:08:25.849970 | controller |         "ip": "172.17.0.5"
2026-03-14 09:08:25.849985 | controller |       },
2026-03-14 09:08:25.850000 | controller |       "storage": {
2026-03-14 09:08:25.850015 | controller |         "ip": "172.18.0.5"
2026-03-14 09:08:25.850030 | controller |       },
2026-03-14 09:08:25.850046 | controller |       "tenant": {
2026-03-14 09:08:25.850062 | controller |         "ip": "172.19.0.5"
2026-03-14 09:08:25.850077 | controller |       }
2026-03-14 09:08:25.850093 | controller |     }
2026-03-14 09:08:25.850109 | controller |   }
2026-03-14 09:08:25.850124 | controller | }
2026-03-14 09:08:25.879319 | controller | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network-instance.yml
2026-03-14 09:08:25.885400 | controller | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network-instance.yml
2026-03-14 09:08:25.897405 | 
2026-03-14 09:08:25.897494 | TASK [Create instance ports and attach if necessary]
2026-03-14 09:08:25.920456 | controller | ok
2026-03-14 09:08:25.977383 | controller | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network-instance-create-attach-port.yml
2026-03-14 09:08:25.984114 | 
2026-03-14 09:08:25.984190 | TASK [Print the host networking data]
2026-03-14 09:08:26.025404 | controller | ok
2026-03-14 09:08:26.030781 | 
2026-03-14 09:08:26.030858 | TASK [Create the private network parent port]
2026-03-14 09:08:28.708354 | controller | admin_state_up: true
2026-03-14 09:08:28.708430 | controller | allowed_address_pairs: []
2026-03-14 09:08:28.708590 | controller | binding_host_id: null
2026-03-14 09:08:28.708725 | controller | binding_profile: {}
2026-03-14 09:08:28.708838 | controller | binding_vif_details: {}
2026-03-14 09:08:28.708942 | controller | binding_vif_type: null
2026-03-14 09:08:28.709071 | controller | binding_vnic_type: normal
2026-03-14 09:08:28.709186 | controller | created_at: '2026-03-14T09:08:28Z'
2026-03-14 09:08:28.709285 | controller | data_plane_status: null
2026-03-14 09:08:28.709373 | controller | description: ''
2026-03-14 09:08:28.709484 | controller | device_id: ''
2026-03-14 09:08:28.709626 | controller | device_owner: ''
2026-03-14 09:08:28.709743 | controller | device_profile: null
2026-03-14 09:08:28.710089 | controller | dns_assignment:
2026-03-14 09:08:28.710186 | controller | - fqdn: host-192-168-122-11.openstacklocal.
2026-03-14 09:08:28.710192 | controller |   hostname: host-192-168-122-11
2026-03-14 09:08:28.710197 | controller |   ip_address: 192.168.122.11
2026-03-14 09:08:28.710204 | controller | dns_domain: ''
2026-03-14 09:08:28.710435 | controller | dns_name: ''
2026-03-14 09:08:28.710552 | controller | extra_dhcp_opts: []
2026-03-14 09:08:28.710819 | controller | fixed_ips:
2026-03-14 09:08:28.710934 | controller | - ip_address: 192.168.122.11
2026-03-14 09:08:28.710940 | controller |   subnet_id: 3293acd3-87a9-4c81-8422-4b391ba88681
2026-03-14 09:08:28.710947 | controller | hardware_offload_type: null
2026-03-14 09:08:28.711024 | controller | hints: ''
2026-03-14 09:08:28.711137 | controller | id: 7613f3f5-a135-4006-852d-a68a6e849070
2026-03-14 09:08:28.711235 | controller | ip_allocation: immediate
2026-03-14 09:08:28.711348 | controller | mac_address: fa:16:3e:23:c7:77
2026-03-14 09:08:28.711477 | controller | name: controller-f396667f-02e9-4d75-b54e-bb319e7c18b8
2026-03-14 09:08:28.711626 | controller | network_id: 993bd7ea-a6c0-4049-a480-d39fe80f9b55
2026-03-14 09:08:28.711740 | controller | numa_affinity_policy: null
2026-03-14 09:08:28.711871 | controller | port_security_enabled: false
2026-03-14 09:08:28.711987 | controller | project_id: 4b633c451ac74233be3721a3635275e5
2026-03-14 09:08:28.712083 | controller | propagate_uplink_status: null
2026-03-14 09:08:28.712353 | controller | resource_request: null
2026-03-14 09:08:28.712472 | controller | revision_number: 1
2026-03-14 09:08:28.712570 | controller | qos_network_policy_id: null
2026-03-14 09:08:28.712700 | controller | qos_policy_id: null
2026-03-14 09:08:28.712799 | controller | security_group_ids: []
2026-03-14 09:08:28.712886 | controller | status: DOWN
2026-03-14 09:08:28.712971 | controller | tags: []
2026-03-14 09:08:28.713066 | controller | trunk_details: null
2026-03-14 09:08:28.713167 | controller | trusted: null
2026-03-14 09:08:28.713276 | controller | updated_at: '2026-03-14T09:08:28Z'
2026-03-14 09:08:29.073995 | controller | changed
2026-03-14 09:08:29.087739 | 
2026-03-14 09:08:29.087891 | TASK [Set common facts for further usage]
2026-03-14 09:08:29.182772 | controller | ok
2026-03-14 09:08:29.196035 | 
2026-03-14 09:08:29.196219 | TASK [Create the trunk port]
2026-03-14 09:08:29.746010 | controller | skipping: Conditional result was False
2026-03-14 09:08:29.762142 | 
2026-03-14 09:08:29.762365 | TASK [Avoid NetworkManager to automatically picked up the new interface]
2026-03-14 09:08:29.840221 | controller | skipping: Conditional result was False
2026-03-14 09:08:29.853335 | 
2026-03-14 09:08:29.853465 | TASK [Refresh NetworkManager]
2026-03-14 09:08:29.920711 | controller | skipping: Conditional result was False
2026-03-14 09:08:29.933419 | 
2026-03-14 09:08:29.933556 | TASK [Attach the port to the instance]
2026-03-14 09:08:36.998283 | controller | changed
2026-03-14 09:08:37.010671 | 
2026-03-14 09:08:37.010836 | TASK [Wait till the interface exists]
2026-03-14 09:08:37.356096 | controller -> 38.102.83.128 | changed
2026-03-14 09:08:37.368862 | 
2026-03-14 09:08:37.369005 | TASK [Prepare interface data to be used configuring trunk and vlan ifaces]
2026-03-14 09:08:37.511280 | controller | ok
2026-03-14 09:08:37.523150 | 
2026-03-14 09:08:37.523279 | LOOP [Create VLANs ports and attach them to the trunk]
2026-03-14 09:08:37.552272 | 
2026-03-14 09:08:37.552518 | LOOP [Prepare VLANs interface data to be used configuring]
2026-03-14 09:08:37.581454 | 
2026-03-14 09:08:37.581738 | LOOP [Create NetworkManager configuration file for the trunk port]
2026-03-14 09:08:38.194870 | controller -> 38.102.83.128 | changed:
2026-03-14 09:08:38.195112 | controller -> 38.102.83.128 | {
2026-03-14 09:08:38.195145 | controller -> 38.102.83.128 |   "key": "default",
2026-03-14 09:08:38.195167 | controller -> 38.102.83.128 |   "value": {
2026-03-14 09:08:38.195184 | controller -> 38.102.83.128 |     "connection": "ci-private-network",
2026-03-14 09:08:38.195200 | controller -> 38.102.83.128 |     "gw": "192.168.122.1",
2026-03-14 09:08:38.195215 | controller -> 38.102.83.128 |     "iface": "eth1",
2026-03-14 09:08:38.195230 | controller -> 38.102.83.128 |     "ip": "192.168.122.11/24",
2026-03-14 09:08:38.195245 | controller -> 38.102.83.128 |     "mac": "fa:16:3e:23:c7:77",
2026-03-14 09:08:38.195260 | controller -> 38.102.83.128 |     "mtu": "1500"
2026-03-14 09:08:38.195278 | controller -> 38.102.83.128 |   }
2026-03-14 09:08:38.195295 | controller -> 38.102.83.128 | }
2026-03-14 09:08:38.195323 | 
2026-03-14 09:08:38.217335 | 
2026-03-14 09:08:38.217528 | TASK [Refresh NetworkManager]
2026-03-14 09:08:39.122451 | controller -> 38.102.83.128 | changed
2026-03-14 09:08:39.140915 | 
2026-03-14 09:08:39.141077 | TASK [Debug fetch IP routes]
2026-03-14 09:08:39.444602 | controller -> 38.102.83.128 | changed
2026-03-14 09:08:39.461076 | 
2026-03-14 09:08:39.461218 | TASK [Debug IP routes]
2026-03-14 09:08:39.498171 | controller -> 38.102.83.128 | ok
2026-03-14 09:08:39.513381 | 
2026-03-14 09:08:39.513533 | TASK [Create instance ports and attach if necessary]
2026-03-14 09:08:39.544046 | controller | ok
2026-03-14 09:08:39.570119 | controller | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network-instance-create-attach-port.yml
2026-03-14 09:08:39.578485 | 
2026-03-14 09:08:39.578570 | TASK [Print the host networking data]
2026-03-14 09:08:39.618284 | controller | ok
2026-03-14 09:08:39.631248 | 
2026-03-14 09:08:39.631392 | TASK [Create the private network parent port]
2026-03-14 09:08:42.989109 | controller | admin_state_up: true
2026-03-14 09:08:42.989194 | controller | allowed_address_pairs: []
2026-03-14 09:08:42.989370 | controller | binding_host_id: null
2026-03-14 09:08:42.989476 | controller | binding_profile: {}
2026-03-14 09:08:42.989577 | controller | binding_vif_details: {}
2026-03-14 09:08:42.989689 | controller | binding_vif_type: null
2026-03-14 09:08:42.989824 | controller | binding_vnic_type: normal
2026-03-14 09:08:42.989957 | controller | created_at: '2026-03-14T09:08:42Z'
2026-03-14 09:08:42.990079 | controller | data_plane_status: null
2026-03-14 09:08:42.990193 | controller | description: ''
2026-03-14 09:08:42.990277 | controller | device_id: ''
2026-03-14 09:08:42.990367 | controller | device_owner: ''
2026-03-14 09:08:42.990462 | controller | device_profile: null
2026-03-14 09:08:42.991632 | controller | dns_assignment:
2026-03-14 09:08:42.993626 | controller | - fqdn: host-192-168-122-10.openstacklocal.
2026-03-14 09:08:42.993649 | controller |   hostname: host-192-168-122-10
2026-03-14 09:08:42.993655 | controller |   ip_address: 192.168.122.10
2026-03-14 09:08:42.993660 | controller | dns_domain: ''
2026-03-14 09:08:42.993664 | controller | dns_name: ''
2026-03-14 09:08:42.993669 | controller | extra_dhcp_opts: []
2026-03-14 09:08:42.993673 | controller | fixed_ips:
2026-03-14 09:08:42.993678 | controller | - ip_address: 192.168.122.10
2026-03-14 09:08:42.993682 | controller |   subnet_id: 3293acd3-87a9-4c81-8422-4b391ba88681
2026-03-14 09:08:42.993687 | controller | hardware_offload_type: null
2026-03-14 09:08:42.993692 | controller | hints: ''
2026-03-14 09:08:42.993701 | controller | id: 41aa7b96-5e47-4b38-affe-d761cc588e1f
2026-03-14 09:08:43.087402 | controller | ip_allocation: immediate
2026-03-14 09:08:43.087458 | controller | mac_address: fa:16:3e:27:7a:64
2026-03-14 09:08:43.087464 | controller | name: crc-7ad45a7c-2b2c-4e9f-b1c1-584bf6854f24
2026-03-14 09:08:43.087470 | controller | network_id: 993bd7ea-a6c0-4049-a480-d39fe80f9b55
2026-03-14 09:08:43.087474 | controller | numa_affinity_policy: null
2026-03-14 09:08:43.087479 | controller | port_security_enabled: false
2026-03-14 09:08:43.087486 | controller | project_id: 4b633c451ac74233be3721a3635275e5
2026-03-14 09:08:43.087491 | controller | propagate_uplink_status: null
2026-03-14 09:08:43.087496 | controller | resource_request: null
2026-03-14 09:08:43.087500 | controller | revision_number: 1
2026-03-14 09:08:43.087504 | controller | qos_network_policy_id: null
2026-03-14 09:08:43.087509 | controller | qos_policy_id: null
2026-03-14 09:08:43.087513 | controller | security_group_ids: []
2026-03-14 09:08:43.087517 | controller | status: DOWN
2026-03-14 09:08:43.087522 | controller | tags: []
2026-03-14 09:08:43.087527 | controller | trunk_details: null
2026-03-14 09:08:43.087531 | controller | trusted: null
2026-03-14 09:08:43.087535 | controller | updated_at: '2026-03-14T09:08:42Z'
2026-03-14 09:08:43.174664 | controller | changed
2026-03-14 09:08:43.188373 | 
2026-03-14 09:08:43.188513 | TASK [Set common facts for further usage]
2026-03-14 09:08:43.273048 | controller | ok
2026-03-14 09:08:43.288218 | 
2026-03-14 09:08:43.288431 | TASK [Create the trunk port]
2026-03-14 09:08:43.841372 | controller | skipping: Conditional result was False
2026-03-14 09:08:43.856066 | 
2026-03-14 09:08:43.856212 | TASK [Avoid NetworkManager to automatically picked up the new interface]
2026-03-14 09:08:43.935480 | controller | skipping: Conditional result was False
2026-03-14 09:08:43.950536 | 
2026-03-14 09:08:43.950710 | TASK [Refresh NetworkManager]
2026-03-14 09:08:44.019228 | controller | skipping: Conditional result was False
2026-03-14 09:08:44.033244 | 
2026-03-14 09:08:44.033385 | TASK [Attach the port to the instance]
2026-03-14 09:08:51.621311 | controller | changed
2026-03-14 09:08:51.633646 | 
2026-03-14 09:08:51.633809 | TASK [Wait till the interface exists]
2026-03-14 09:08:52.031109 | controller -> 38.102.83.162 | changed
2026-03-14 09:08:52.049165 | 
2026-03-14 09:08:52.049441 | TASK [Prepare interface data to be used configuring trunk and vlan ifaces]
2026-03-14 09:08:52.195637 | controller | ok
2026-03-14 09:08:52.206507 | 
2026-03-14 09:08:52.206711 | LOOP [Create VLANs ports and attach them to the trunk]
2026-03-14 09:08:52.274853 | controller | skipping: Conditional result was False
2026-03-14 09:08:55.280402 | controller | skipping: Conditional result was False
2026-03-14 09:08:58.296575 | controller | skipping: Conditional result was False
2026-03-14 09:08:58.310232 | 
2026-03-14 09:08:58.310428 | LOOP [Prepare VLANs interface data to be used configuring]
2026-03-14 09:08:58.453929 | controller | ok:
2026-03-14 09:08:58.454453 | controller | {
2026-03-14 09:08:58.454530 | controller |   "key": "internal-api",
2026-03-14 09:08:58.454575 | controller |   "value": {
2026-03-14 09:08:58.454614 | controller |     "ip": "172.17.0.5"
2026-03-14 09:08:58.454652 | controller |   }
2026-03-14 09:08:58.454728 | controller | }
2026-03-14 09:08:58.454798 | 
2026-03-14 09:08:58.478202 | controller | ok:
2026-03-14 09:08:58.478304 | controller | {
2026-03-14 09:08:58.478353 | controller |   "key": "storage",
2026-03-14 09:08:58.478392 | controller |   "value": {
2026-03-14 09:08:58.478428 | controller |     "ip": "172.18.0.5"
2026-03-14 09:08:58.478464 | controller |   }
2026-03-14 09:08:58.478498 | controller | }
2026-03-14 09:08:58.508074 | controller | ok:
2026-03-14 09:08:58.508189 | controller | {
2026-03-14 09:08:58.508248 | controller |   "key": "tenant",
2026-03-14 09:08:58.508295 | controller |   "value": {
2026-03-14 09:08:58.508342 | controller |     "ip": "172.19.0.5"
2026-03-14 09:08:58.508385 | controller |   }
2026-03-14 09:08:58.508426 | controller | }
2026-03-14 09:08:58.529942 | 
2026-03-14 09:08:58.530101 | LOOP [Create NetworkManager configuration file for the trunk port]
2026-03-14 09:08:59.474248 | controller -> 38.102.83.162 | changed:
2026-03-14 09:08:59.474650 | controller -> 38.102.83.162 | {
2026-03-14 09:08:59.474770 | controller -> 38.102.83.162 |   "key": "default",
2026-03-14 09:08:59.474825 | controller -> 38.102.83.162 |   "value": {
2026-03-14 09:08:59.474870 | controller -> 38.102.83.162 |     "connection": "ci-private-network",
2026-03-14 09:08:59.474915 | controller -> 38.102.83.162 |     "gw": "192.168.122.1",
2026-03-14 09:08:59.474956 | controller -> 38.102.83.162 |     "iface": "ens7",
2026-03-14 09:08:59.474999 | controller -> 38.102.83.162 |     "ip": "192.168.122.10/24",
2026-03-14 09:08:59.475039 | controller -> 38.102.83.162 |     "mac": "fa:16:3e:27:7a:64",
2026-03-14 09:08:59.475078 | controller -> 38.102.83.162 |     "mtu": "1500"
2026-03-14 09:08:59.475119 | controller -> 38.102.83.162 |   }
2026-03-14 09:08:59.475160 | controller -> 38.102.83.162 | }
2026-03-14 09:09:00.373305 | controller -> 38.102.83.162 | changed:
2026-03-14 09:09:00.373471 | controller -> 38.102.83.162 | {
2026-03-14 09:09:00.373520 | controller -> 38.102.83.162 |   "key": "internal-api",
2026-03-14 09:09:00.373558 | controller -> 38.102.83.162 |   "value": {
2026-03-14 09:09:00.373594 | controller -> 38.102.83.162 |     "connection": "ci-private-network-20",
2026-03-14 09:09:00.373629 | controller -> 38.102.83.162 |     "iface": "ens7.20",
2026-03-14 09:09:00.373661 | controller -> 38.102.83.162 |     "ip": "172.17.0.5/24",
2026-03-14 09:09:00.373734 | controller -> 38.102.83.162 |     "mac": "52:54:00:f1:69:be",
2026-03-14 09:09:00.373771 | controller -> 38.102.83.162 |     "mtu": "1496",
2026-03-14 09:09:00.373804 | controller -> 38.102.83.162 |     "parent_iface": "ens7",
2026-03-14 09:09:00.373836 | controller -> 38.102.83.162 |     "vlan": 20
2026-03-14 09:09:00.373901 | controller -> 38.102.83.162 |   }
2026-03-14 09:09:00.373937 | controller -> 38.102.83.162 | }
2026-03-14 09:09:01.330436 | controller -> 38.102.83.162 | changed:
2026-03-14 09:09:01.330636 | controller -> 38.102.83.162 | {
2026-03-14 09:09:01.330804 | controller -> 38.102.83.162 |   "key": "storage",
2026-03-14 09:09:01.330897 | controller -> 38.102.83.162 |   "value": {
2026-03-14 09:09:01.330977 | controller -> 38.102.83.162 |     "connection": "ci-private-network-21",
2026-03-14 09:09:01.331052 | controller -> 38.102.83.162 |     "iface": "ens7.21",
2026-03-14 09:09:01.331123 | controller -> 38.102.83.162 |     "ip": "172.18.0.5/24",
2026-03-14 09:09:01.331191 | controller -> 38.102.83.162 |     "mac": "52:54:00:fb:70:a1",
2026-03-14 09:09:01.331257 | controller -> 38.102.83.162 |     "mtu": "1496",
2026-03-14 09:09:01.331330 | controller -> 38.102.83.162 |     "parent_iface": "ens7",
2026-03-14 09:09:01.331403 | controller -> 38.102.83.162 |     "vlan": 21
2026-03-14 09:09:01.331474 | controller -> 38.102.83.162 |   }
2026-03-14 09:09:01.331534 | controller -> 38.102.83.162 | }
2026-03-14 09:09:02.245545 | controller -> 38.102.83.162 | changed:
2026-03-14 09:09:02.245773 | controller -> 38.102.83.162 | {
2026-03-14 09:09:02.245828 | controller -> 38.102.83.162 |   "key": "tenant",
2026-03-14 09:09:02.245866 | controller -> 38.102.83.162 |   "value": {
2026-03-14 09:09:02.245902 | controller -> 38.102.83.162 |     "connection": "ci-private-network-22",
2026-03-14 09:09:02.245936 | controller -> 38.102.83.162 |     "iface": "ens7.22",
2026-03-14 09:09:02.245972 | controller -> 38.102.83.162 |     "ip": "172.19.0.5/24",
2026-03-14 09:09:02.246006 | controller -> 38.102.83.162 |     "mac": "52:54:00:b4:31:ab",
2026-03-14 09:09:02.246042 | controller -> 38.102.83.162 |     "mtu": "1496",
2026-03-14 09:09:02.246076 | controller -> 38.102.83.162 |     "parent_iface": "ens7",
2026-03-14 09:09:02.246109 | controller -> 38.102.83.162 |     "vlan": 22
2026-03-14 09:09:02.246143 | controller -> 38.102.83.162 |   }
2026-03-14 09:09:02.246177 | controller -> 38.102.83.162 | }
2026-03-14 09:09:02.269625 | 
2026-03-14 09:09:02.269921 | TASK [Refresh NetworkManager]
2026-03-14 09:09:03.157792 | controller -> 38.102.83.162 | changed
2026-03-14 09:09:03.179479 | 
2026-03-14 09:09:03.179652 | TASK [Debug fetch IP routes]
2026-03-14 09:09:03.523257 | controller -> 38.102.83.162 | changed
2026-03-14 09:09:03.532395 | 
2026-03-14 09:09:03.532484 | TASK [Debug IP routes]
2026-03-14 09:09:03.588858 | controller -> 38.102.83.162 | ok
2026-03-14 09:09:03.598049 | 
2026-03-14 09:09:03.598139 | TASK [Display some data about network ports]
2026-03-14 09:09:05.362303 | controller | +--------------------------------------+-------------------------------------------------+-------------------+-------------------------------------------------------------------------------+--------+
2026-03-14 09:09:05.478022 | controller | | ID                                   | Name                                            | MAC Address       | Fixed IP Addresses                                                            | Status |
2026-03-14 09:09:05.478092 | controller | +--------------------------------------+-------------------------------------------------+-------------------+-------------------------------------------------------------------------------+--------+
2026-03-14 09:09:05.478105 | controller | | 41aa7b96-5e47-4b38-affe-d761cc588e1f | crc-7ad45a7c-2b2c-4e9f-b1c1-584bf6854f24        | fa:16:3e:27:7a:64 | ip_address='192.168.122.10', subnet_id='3293acd3-87a9-4c81-8422-4b391ba88681' | ACTIVE |
2026-03-14 09:09:05.478115 | controller | | 7613f3f5-a135-4006-852d-a68a6e849070 | controller-f396667f-02e9-4d75-b54e-bb319e7c18b8 | fa:16:3e:23:c7:77 | ip_address='192.168.122.11', subnet_id='3293acd3-87a9-4c81-8422-4b391ba88681' | ACTIVE |
2026-03-14 09:09:05.478124 | controller | | d1476c2e-8ba7-462e-b768-cd523390318f |                                                 | fa:16:3e:66:ca:76 |                                                                               | DOWN   |
2026-03-14 09:09:05.478134 | controller | +--------------------------------------+-------------------------------------------------+-------------------+-------------------------------------------------------------------------------+--------+
2026-03-14 09:09:05.642454 | controller | changed
2026-03-14 09:09:05.655543 | 
2026-03-14 09:09:05.655746 | LOOP [Display server configuration]
2026-03-14 09:09:08.411828 | controller | changed
2026-03-14 09:09:08.412104 | controller | changed: All items complete
2026-03-14 09:09:08.412146 | 
2026-03-14 09:09:14.695859 | controller | changed
2026-03-14 09:09:14.709816 | 
2026-03-14 09:09:14.709983 | TASK [Fetch underneath provider DNSs]
2026-03-14 09:09:14.736375 | controller | ok
2026-03-14 09:09:14.746536 | controller | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network-fetch-provider-dns.yml
2026-03-14 09:09:14.763522 | 
2026-03-14 09:09:14.763632 | TASK [Fetch the DNS list of the default interface]
2026-03-14 09:09:15.168070 | controller | ok
2026-03-14 09:09:15.181998 | 
2026-03-14 09:09:15.182133 | TASK [Set provider DNS for further usage]
2026-03-14 09:09:15.255177 | controller | ok
2026-03-14 09:09:15.266722 | 
2026-03-14 09:09:15.266855 | LOOP [Save networking data to file for further usage]
2026-03-14 09:09:16.328120 | controller -> 38.102.83.162 | changed: "crc"
2026-03-14 09:09:16.900572 | controller -> 38.102.83.128 | changed: "controller"
2026-03-14 09:09:16.919076 | 
2026-03-14 09:09:16.919285 | TASK [Remove cloud_secrets file]
2026-03-14 09:09:17.152977 | controller | changed
2026-03-14 09:09:17.195403 | 
2026-03-14 09:09:17.195510 | PLAY RECAP
2026-03-14 09:09:17.195582 | controller | ok: 48 changed: 22 unreachable: 0 failed: 0 skipped: 14 rescued: 0 ignored: 0
2026-03-14 09:09:17.195619 | 
2026-03-14 09:09:17.312278 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network.yml@master]
2026-03-14 09:09:17.322946 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/prepare-crc.yaml@master]
2026-03-14 09:09:17.966172 | 
2026-03-14 09:09:17.966291 | PLAY [Prefer ipv4 over ipv6]
2026-03-14 09:09:18.003621 | 
2026-03-14 09:09:18.003788 | TASK [Configure /etc/gai.conf to prever ipv4 over ipv6]
2026-03-14 09:09:18.070508 | crc | skipping: Conditional result was False
2026-03-14 09:09:18.071001 | controller | skipping: Conditional result was False
2026-03-14 09:09:18.118008 | 
2026-03-14 09:09:18.118119 | PLAY [Generic task for migration time to crc-cloud]
2026-03-14 09:09:18.138027 | 
2026-03-14 09:09:18.138110 | TASK [Check if there is a image build date file]
2026-03-14 09:09:18.593502 | crc | ok
2026-03-14 09:09:18.610025 | 
2026-03-14 09:09:18.610191 | TASK [Print image date]
2026-03-14 09:09:19.116891 | crc | Sat Mar 14 02:51:45 UTC 2026 | Image type: crc-cloud
2026-03-14 09:09:19.201386 | crc | ok
2026-03-14 09:09:19.215800 | 
2026-03-14 09:09:19.215957 | TASK [Run crc-cloud preparation]
2026-03-14 09:09:19.259633 | crc | ok
2026-03-14 09:09:19.303179 | 
2026-03-14 09:09:19.303313 | TASK [prepare-crc-cloud : Print build date]
2026-03-14 09:09:19.338831 | crc | ok
2026-03-14 09:09:19.350026 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/build_date.yaml
2026-03-14 09:09:19.363131 | 
2026-03-14 09:09:19.363216 | TASK [prepare-crc-cloud : Check if there is a image build date file]
2026-03-14 09:09:19.698586 | crc | ok
2026-03-14 09:09:19.714957 | 
2026-03-14 09:09:19.715096 | TASK [prepare-crc-cloud : Print image date if available]
2026-03-14 09:09:20.087979 | crc | Sat Mar 14 02:51:45 UTC 2026 | Image type: crc-cloud
2026-03-14 09:09:20.271185 | crc | ok
2026-03-14 09:09:20.296117 | 
2026-03-14 09:09:20.296302 | TASK [prepare-crc-cloud : Tune Kubelet config file]
2026-03-14 09:09:20.323908 | crc | skipping: Conditional result was False
2026-03-14 09:09:20.338917 | 
2026-03-14 09:09:20.339056 | TASK [prepare-crc-cloud : Reboot CRC host to get the network interfaces]
2026-03-14 09:09:20.374847 | crc | ok
2026-03-14 09:09:20.386008 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/reboot_crc.yaml
2026-03-14 09:09:20.397217 | 
2026-03-14 09:09:20.397315 | TASK [prepare-crc-cloud : Reboot host after adding pull secret and changing kubelet node params]
2026-03-14 09:09:59.398725 | crc | changed
2026-03-14 09:09:59.413654 | 
2026-03-14 09:09:59.413866 | TASK [Start Zuul console after reboot]
2026-03-14 09:09:59.458978 | crc | ok
2026-03-14 09:09:59.498922 | 
2026-03-14 09:09:59.499055 | TASK [start-zuul-console : Start zuul_console daemon.]
2026-03-14 09:10:00.153088 | crc | ok
2026-03-14 09:10:00.184652 | 
2026-03-14 09:10:00.184867 | TASK [prepare-crc-cloud : Prepare for executing crc-cloud]
2026-03-14 09:10:00.221047 | crc | ok
2026-03-14 09:10:00.233372 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/pre.yaml
2026-03-14 09:10:00.250708 | 
2026-03-14 09:10:00.250816 | TASK [prepare-crc-cloud : Disable overwriting /etc/resolv.conf by the NetworkManager]
2026-03-14 09:10:01.371168 | crc | changed
2026-03-14 09:10:01.388643 | 
2026-03-14 09:10:01.388812 | TASK [prepare-crc-cloud : Reload NetworkManager after creating custom rules]
2026-03-14 09:10:02.298195 | crc | changed
2026-03-14 09:10:02.320900 | 
2026-03-14 09:10:02.321080 | TASK [prepare-crc-cloud : Inject pull secret into /var/lib/kubelet/config.json]
2026-03-14 09:10:03.386371 | crc | changed
2026-03-14 09:10:03.405887 | 
2026-03-14 09:10:03.406050 | TASK [prepare-crc-cloud : Check if etcd slow profile set]
2026-03-14 09:10:03.433503 | crc | skipping: Conditional result was False
2026-03-14 09:10:03.454740 | 
2026-03-14 09:10:03.454949 | TASK [prepare-crc-cloud : Use ramdisk for etcd]
2026-03-14 09:10:03.535614 | crc | ok
2026-03-14 09:10:03.555450 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/ramdisk.yaml
2026-03-14 09:10:03.571749 | 
2026-03-14 09:10:03.571858 | TASK [prepare-crc-cloud : Move current etcd to other place]
2026-03-14 09:10:04.110256 | crc | ok
2026-03-14 09:10:04.126640 | 
2026-03-14 09:10:04.126807 | TASK [prepare-crc-cloud : Mount ramdisk]
2026-03-14 09:10:04.601549 | crc | changed
2026-03-14 09:10:04.618598 | 
2026-03-14 09:10:04.618788 | TASK [prepare-crc-cloud : Set proper permissions after mount]
2026-03-14 09:10:05.110792 | crc | changed
2026-03-14 09:10:05.127885 | 
2026-03-14 09:10:05.128031 | TASK [prepare-crc-cloud : Copy content from old location to ramdisk]
2026-03-14 09:10:06.564657 | crc | changed
2026-03-14 09:10:06.574917 | 
2026-03-14 09:10:06.575038 | TASK [prepare-crc-cloud : Set proper SELinux context]
2026-03-14 09:10:07.119446 | crc | ok
2026-03-14 09:10:07.136241 | 
2026-03-14 09:10:07.136431 | TASK [prepare-crc-cloud : Block crc.testing domains]
2026-03-14 09:10:07.180967 | crc | ok
2026-03-14 09:10:07.202606 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/block_crc_domains.yaml
2026-03-14 09:10:07.226672 | 
2026-03-14 09:10:07.226820 | LOOP [prepare-crc-cloud : Set crc.testing domains into /etc/hosts]
2026-03-14 09:10:07.684566 | crc | changed: line added
2026-03-14 09:10:07.955601 | crc | changed: line added
2026-03-14 09:10:08.223467 | crc | changed: line added
2026-03-14 09:10:08.514389 | crc | changed: line added
2026-03-14 09:10:08.783005 | crc | changed: line added
2026-03-14 09:10:09.062233 | crc | changed: line added
2026-03-14 09:10:09.368372 | crc | changed: line added
2026-03-14 09:10:09.653583 | crc | changed: line added
2026-03-14 09:10:09.677042 | 
2026-03-14 09:10:09.677190 | TASK [prepare-crc-cloud : Get default interface name]
2026-03-14 09:10:10.064425 | crc | ens3
2026-03-14 09:10:10.268081 | crc | ok
2026-03-14 09:10:10.284311 | 
2026-03-14 09:10:10.284450 | LOOP [prepare-crc-cloud : Block CRC domains also using iptables]
2026-03-14 09:10:10.844718 | crc | ok
2026-03-14 09:10:10.844994 | crc | changed: All items complete
2026-03-14 09:10:10.845032 | 
2026-03-14 09:10:11.190067 | crc | ok
2026-03-14 09:10:11.519432 | crc | ok
2026-03-14 09:10:11.864595 | crc | ok
2026-03-14 09:10:12.196571 | crc | ok
2026-03-14 09:10:12.523870 | crc | ok
2026-03-14 09:10:12.851844 | crc | ok
2026-03-14 09:10:13.161867 | crc | ok
2026-03-14 09:10:13.191074 | 
2026-03-14 09:10:13.191275 | TASK [prepare-crc-cloud : Set required facts]
2026-03-14 09:10:13.224152 | crc | ok
2026-03-14 09:10:13.236643 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/set_vars.yaml
2026-03-14 09:10:13.261346 | 
2026-03-14 09:10:13.261484 | TASK [prepare-crc-cloud : Set required facts]
2026-03-14 09:10:13.324878 | crc | Output suppressed because no_log was given
2026-03-14 09:10:13.339263 | 
2026-03-14 09:10:13.339400 | TASK [Start crc-cloud role when Zuul defined]
2026-03-14 09:10:13.374853 | crc | ok
2026-03-14 09:10:13.435369 | 
2026-03-14 09:10:13.435490 | TASK [deploy-crc-cloud : Check if pull-secret is provided]
2026-03-14 09:10:13.459575 | crc | skipping: Conditional result was False
2026-03-14 09:10:13.468411 | 
2026-03-14 09:10:13.468480 | TASK [deploy-crc-cloud : Create kubeconfig]
2026-03-14 09:10:13.499743 | crc | ok
2026-03-14 09:10:13.508045 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/kubeconfig.yaml
2026-03-14 09:10:13.520846 | 
2026-03-14 09:10:13.520943 | TASK [deploy-crc-cloud : Create kube directory]
2026-03-14 09:10:13.786100 | crc | changed
2026-03-14 09:10:13.793191 | 
2026-03-14 09:10:13.793259 | TASK [deploy-crc-cloud : Copy kubeconfig to user dir]
2026-03-14 09:10:14.283244 | crc | changed
2026-03-14 09:10:14.298887 | 
2026-03-14 09:10:14.299029 | TASK [deploy-crc-cloud : Setup dnsmasq]
2026-03-14 09:10:14.341400 | crc | ok
2026-03-14 09:10:14.362378 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/dnsmasq.yaml
2026-03-14 09:10:14.381497 | 
2026-03-14 09:10:14.381615 | TASK [deploy-crc-cloud : Disable overwriting /etc/resolv.conf by the NetworkManager]
2026-03-14 09:10:15.302776 | crc | ok
2026-03-14 09:10:15.312960 | 
2026-03-14 09:10:15.313060 | TASK [deploy-crc-cloud : Restart NetworkManager when its needed]
2026-03-14 09:10:15.338661 | crc | skipping: Conditional result was False
2026-03-14 09:10:15.389964 | 
2026-03-14 09:10:15.390088 | TASK [deploy-crc-cloud : Ensure /etc/resolv.conf is not a symlink and is writable]
2026-03-14 09:10:15.699516 | crc | ok
2026-03-14 09:10:15.709625 | 
2026-03-14 09:10:15.709743 | TASK [deploy-crc-cloud : Create crc-dnsmasq.conf]
2026-03-14 09:10:16.663917 | crc | changed
2026-03-14 09:10:16.682169 | 
2026-03-14 09:10:16.682342 | TASK [deploy-crc-cloud : Add domains into crc-dnsmasq.conf when alternative_domain]
2026-03-14 09:10:16.709617 | crc | skipping: Conditional result was False
2026-03-14 09:10:16.721146 | 
2026-03-14 09:10:16.721257 | LOOP [deploy-crc-cloud : Set this host as first nameserver in /etc/resolv.conf]
2026-03-14 09:10:17.074261 | crc | changed: line replaced
2026-03-14 09:10:17.358267 | crc | ok
2026-03-14 09:10:17.651531 | crc | ok
2026-03-14 09:10:17.676885 | 
2026-03-14 09:10:17.677098 | TASK [deploy-crc-cloud : Restart dnsmasq]
2026-03-14 09:10:18.628197 | crc | changed
2026-03-14 09:10:18.648456 | 
2026-03-14 09:10:18.648594 | TASK [deploy-crc-cloud : Start kubelet]
2026-03-14 09:10:18.691674 | crc | ok
2026-03-14 09:10:18.711490 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/kubelet.yaml
2026-03-14 09:10:18.736790 | 
2026-03-14 09:10:18.736932 | TASK [deploy-crc-cloud : Start and enable kubelet]
2026-03-14 09:10:33.336353 | crc | changed
2026-03-14 09:10:33.356820 | 
2026-03-14 09:10:33.356953 | TASK [deploy-crc-cloud : Wait for port 6443 to be up]
2026-03-14 09:11:33.830106 | crc | ok
2026-03-14 09:11:33.846097 | 
2026-03-14 09:11:33.846417 | TASK [deploy-crc-cloud : Wait for API to start before continue]
2026-03-14 09:11:35.130855 | crc | NAMESPACE                         NAME                                         READY   STATUS    RESTARTS   AGE
2026-03-14 09:11:35.130941 | crc | openshift-network-console         networking-console-plugin-85b44fc459-gdk6g   1/1     Running   2          384d
2026-03-14 09:11:35.130966 | crc | openshift-network-diagnostics     network-check-source-55646444c4-trplf        1/1     Running   2          384d
2026-03-14 09:11:35.136313 | crc | openshift-network-diagnostics     network-check-target-xd92c                   1/1     Running   2          384d
2026-03-14 09:11:35.136513 | crc | openshift-network-node-identity   network-node-identity-vrzqb                  2/2     Running   7          384d
2026-03-14 09:11:35.136535 | crc | openshift-network-operator        iptables-alerter-4ln5h                       1/1     Running   2          384d
2026-03-14 09:11:35.136549 | crc | openshift-network-operator        network-operator-58b4c7f79c-55gtf            1/1     Running   4          384d
2026-03-14 09:11:35.406539 | crc | ok
2026-03-14 09:11:35.422754 | 
2026-03-14 09:11:35.422984 | TASK [deploy-crc-cloud : Replace default pubkey]
2026-03-14 09:11:35.467344 | crc | ok
2026-03-14 09:11:35.489241 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/pubkey.yaml
2026-03-14 09:11:35.517322 | 
2026-03-14 09:11:35.517464 | TASK [deploy-crc-cloud : Check if id_rsa.pub exists]
2026-03-14 09:11:35.845364 | crc | ok
2026-03-14 09:11:35.862847 | 
2026-03-14 09:11:35.862985 | TASK [deploy-crc-cloud : Read pub key]
2026-03-14 09:11:36.421033 | crc | skipping: Conditional result was False
2026-03-14 09:11:36.439124 | 
2026-03-14 09:11:36.439288 | TASK [deploy-crc-cloud : Wait for machineconfig]
2026-03-14 09:11:36.467402 | crc | skipping: Conditional result was False
2026-03-14 09:11:36.485513 | 
2026-03-14 09:11:36.485703 | TASK [deploy-crc-cloud : Patch machineconfig 99-master-ssh]
2026-03-14 09:11:37.071910 | crc | skipping: Conditional result was False
2026-03-14 09:11:37.090775 | 
2026-03-14 09:11:37.090960 | TASK [deploy-crc-cloud : Accept certificate]
2026-03-14 09:11:37.176394 | crc | ok
2026-03-14 09:11:37.197153 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/accept_cert.yaml
2026-03-14 09:11:37.229156 | 
2026-03-14 09:11:37.229260 | LOOP [deploy-crc-cloud : Re-run accept certificates in Pending state few times]
2026-03-14 09:11:37.769513 | crc | ok
2026-03-14 09:11:37.769922 | crc | changed: All items complete
2026-03-14 09:11:37.769984 | 
2026-03-14 09:11:48.388151 | crc | ok
2026-03-14 09:11:58.861912 | crc | ok
2026-03-14 09:11:58.886358 | 
2026-03-14 09:11:58.886554 | TASK [deploy-crc-cloud : Create service account for - auto-csr-approver]
2026-03-14 09:11:59.727933 | crc | changed
2026-03-14 09:11:59.744653 | 
2026-03-14 09:11:59.744855 | TASK [deploy-crc-cloud : Apply cluster cert approver]
2026-03-14 09:12:00.212960 | crc | serviceaccount/csr-approver-sa created
2026-03-14 09:12:00.218573 | crc | clusterrole.rbac.authorization.k8s.io/csr-approver created
2026-03-14 09:12:00.228771 | crc | clusterrolebinding.rbac.authorization.k8s.io/csr-approver-binding created
2026-03-14 09:12:00.240555 | crc | cronjob.batch/auto-csr-approver created
2026-03-14 09:12:00.305173 | crc | ok
2026-03-14 09:12:00.315382 | 
2026-03-14 09:12:00.315473 | TASK [deploy-crc-cloud : Wait for cluster become healthy]
2026-03-14 09:12:00.350975 | crc | ok
2026-03-14 09:12:00.363224 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/wait_cluster_become_healthy.yaml
2026-03-14 09:12:00.395775 | 
2026-03-14 09:12:00.395846 | TASK [deploy-crc-cloud : Check if specified OpenShift components are healthy etcd|openshift-apiserver]
2026-03-14 09:12:00.905879 | crc | True
2026-03-14 09:12:00.907065 | crc | False
2026-03-14 09:12:47.502252 | crc | changed
2026-03-14 09:12:47.513339 | 
2026-03-14 09:12:47.513445 | TASK [deploy-crc-cloud : Output success message if components are healthy]
2026-03-14 09:12:47.561590 | crc | ok: OpenShift cluster components have become healthy in approximately 60 * 15 seconds.
2026-03-14 09:12:47.571897 | 
2026-03-14 09:12:47.572004 | TASK [deploy-crc-cloud : Fail if any components are still unhealthy]
2026-03-14 09:12:47.588223 | crc | skipping: Conditional result was False
2026-03-14 09:12:47.601907 | 
2026-03-14 09:12:47.602048 | TASK [deploy-crc-cloud : Set credentials]
2026-03-14 09:12:47.628128 | crc | ok
2026-03-14 09:12:47.644576 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/set_credentials.yaml
2026-03-14 09:12:47.675369 | 
2026-03-14 09:12:47.675486 | TASK [deploy-crc-cloud : Ensure htpasswd.txt does not exists]
2026-03-14 09:12:47.964815 | crc | ok
2026-03-14 09:12:47.982084 | 
2026-03-14 09:12:47.982240 | TASK [deploy-crc-cloud : Create temporary directory]
2026-03-14 09:12:48.424274 | crc | changed
2026-03-14 09:12:48.442391 | 
2026-03-14 09:12:48.442528 | LOOP [deploy-crc-cloud : Get htpasswd]
2026-03-14 09:12:48.529044 | crc | Output suppressed because no_log was given
2026-03-14 09:12:48.549027 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/get_htpasswd.yaml
2026-03-14 09:12:48.550368 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/get_htpasswd.yaml
2026-03-14 09:12:48.551638 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/get_htpasswd.yaml
2026-03-14 09:12:48.566603 | 
2026-03-14 09:12:48.566757 | TASK [deploy-crc-cloud : Get htpasswd for developer]
2026-03-14 09:13:08.826908 | crc | time="2026-03-14T09:13:08Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
2026-03-14 09:13:08.829400 | crc | Trying to pull ghcr.io/crc-org/crc-cloud/httpd-tools:latest...
2026-03-14 09:13:09.171331 | crc | Getting image source signatures
2026-03-14 09:13:12.157444 | crc | Copying blob sha256:1730cf1a00b8c06596486f3456376e7660a42976bb9fcd6604ae22a70fca6b65
2026-03-14 09:13:12.157496 | crc | Copying blob sha256:f391d299bb65afd981883c104f434a04ee3ffb247e341247171779ef526ed65f
2026-03-14 09:13:12.157511 | crc | Copying config sha256:d81064f8940a06af9e1a39fc510c6bfc299bb166f99ee6a892f22ec84ee58041
2026-03-14 09:13:12.167161 | crc | Writing manifest to image destination
2026-03-14 09:13:12.699762 | crc | changed
2026-03-14 09:13:12.717575 | 
2026-03-14 09:13:12.717744 | TASK [deploy-crc-cloud : Get htpasswd for kubeadmin]
2026-03-14 09:13:13.105866 | crc | time="2026-03-14T09:13:13Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
2026-03-14 09:13:13.775465 | crc | changed
2026-03-14 09:13:13.787050 | 
2026-03-14 09:13:13.787151 | TASK [deploy-crc-cloud : Get htpasswd for redhat]
2026-03-14 09:13:14.231928 | crc | time="2026-03-14T09:13:14Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
2026-03-14 09:13:15.362896 | crc | changed
2026-03-14 09:13:15.378856 | 
2026-03-14 09:13:15.379019 | TASK [deploy-crc-cloud : Cleanup htpasswd.txt file]
2026-03-14 09:13:15.945082 | crc | changed
2026-03-14 09:13:15.961527 | 
2026-03-14 09:13:15.961715 | TASK [deploy-crc-cloud : Create secret with generic htpass-secret]
2026-03-14 09:13:16.557398 | crc | ok
2026-03-14 09:13:16.573524 | 
2026-03-14 09:13:16.573662 | TASK [deploy-crc-cloud : Replace htpass-secret]
2026-03-14 09:13:17.055216 | crc | secret/htpass-secret replaced
2026-03-14 09:13:17.154808 | crc | ok
2026-03-14 09:13:17.171321 | 
2026-03-14 09:13:17.171455 | TASK [deploy-crc-cloud : Remove temporary directory]
2026-03-14 09:13:17.583645 | crc | changed
2026-03-14 09:13:17.600921 | 
2026-03-14 09:13:17.601061 | TASK [deploy-crc-cloud : Replace default CA]
2026-03-14 09:13:17.643540 | crc | ok
2026-03-14 09:13:17.667088 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/replace_default_ca.yaml
2026-03-14 09:13:17.701511 | 
2026-03-14 09:13:17.701576 | TASK [deploy-crc-cloud : Generate key]
2026-03-14 09:13:19.255475 | crc | changed
2026-03-14 09:13:19.273025 | 
2026-03-14 09:13:19.273302 | TASK [deploy-crc-cloud : Generate cert]
2026-03-14 09:13:19.841663 | crc | changed
2026-03-14 09:13:19.852434 | 
2026-03-14 09:13:19.852536 | TASK [deploy-crc-cloud : Generate csr]
2026-03-14 09:13:20.348427 | crc | ..+.+..+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+...+.........+.+.........+..............+.+...+..+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+.......+...+............+......+.....+.......+..+.+............+..+................+..+...+.........+....+.....+.+......+...+..+...+.......+............+......+..........................+...+...+....+...+...............+.....+.+...........+....+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2026-03-14 09:13:20.410021 | crc | ....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+................+...+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.+......+........+....+...+.....+.......+.....+............................+..+...+....+.........+.....+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2026-03-14 09:13:20.415345 | crc | -----
2026-03-14 09:13:20.902606 | crc | changed
2026-03-14 09:13:20.914565 | 
2026-03-14 09:13:20.914707 | TASK [deploy-crc-cloud : Generate user cert]
2026-03-14 09:13:21.290068 | crc | Certificate request self-signature ok
2026-03-14 09:13:21.303124 | crc | subject=O = ${GROUP}, CN = core
2026-03-14 09:13:21.507795 | crc | changed
2026-03-14 09:13:21.524122 | 
2026-03-14 09:13:21.524280 | TASK [deploy-crc-cloud : Create configmap]
2026-03-14 09:13:22.126988 | crc | configmap/client-ca-custom created
2026-03-14 09:13:22.594059 | crc | ok
2026-03-14 09:13:22.613447 | 
2026-03-14 09:13:22.613608 | TASK [deploy-crc-cloud : Patch apiserver]
2026-03-14 09:13:23.090374 | crc | apiserver.config.openshift.io/cluster patched
2026-03-14 09:13:23.192077 | crc | ok
2026-03-14 09:13:23.209332 | 
2026-03-14 09:13:23.209516 | TASK [deploy-crc-cloud : Create configmap]
2026-03-14 09:13:23.708312 | crc | W0314 09:13:23.708211   12884 helpers.go:703] --dry-run is deprecated and can be replaced with --dry-run=client.
2026-03-14 09:13:23.767865 | crc | configmap/admin-kubeconfig-client-ca replaced
2026-03-14 09:13:24.282970 | crc | ok
2026-03-14 09:13:24.293598 | 
2026-03-14 09:13:24.293761 | TASK [deploy-crc-cloud : Login to the OpenShift cluster]
2026-03-14 09:13:24.329630 | crc | ok
2026-03-14 09:13:24.340631 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/login.yaml
2026-03-14 09:13:24.372887 | 
2026-03-14 09:13:24.372999 | TASK [deploy-crc-cloud : Try to login after all changes]
2026-03-14 09:13:24.885257 | crc | WARNING: Using insecure TLS client config. Setting this option is not supported!
2026-03-14 09:13:24.930103 | crc |
2026-03-14 09:13:24.930173 | crc | Unable to connect to the server: EOF
2026-03-14 09:14:42.419308 | crc | ok
2026-03-14 09:14:42.435981 | 
2026-03-14 09:14:42.436126 | TASK [deploy-crc-cloud : Patch pull secret]
2026-03-14 09:14:42.479882 | crc | ok
2026-03-14 09:14:42.496502 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/patch_pull_secret.yaml
2026-03-14 09:14:42.536332 | 
2026-03-14 09:14:42.536448 | TASK [deploy-crc-cloud : Patch pull secret]
2026-03-14 09:14:43.077208 | crc | Output suppressed because no_log was given
2026-03-14 09:14:43.092218 | 
2026-03-14 09:14:43.092358 | TASK [deploy-crc-cloud : Wait for cluster become healthy after patching CA and pull secret]
2026-03-14 09:14:43.134811 | crc | ok
2026-03-14 09:14:43.153280 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/wait_cluster_become_healthy.yaml
2026-03-14 09:14:43.193984 | 
2026-03-14 09:14:43.194097 | TASK [deploy-crc-cloud : Check if specified OpenShift components are healthy etcd|openshift-apiserver]
2026-03-14 09:14:43.680266 | crc | True
2026-03-14 09:14:43.681039 | crc | True
2026-03-14 09:14:43.732092 | crc | changed
2026-03-14 09:14:43.749609 | 
2026-03-14 09:14:43.749785 | TASK [deploy-crc-cloud : Output success message if components are healthy]
2026-03-14 09:14:43.807001 | crc | ok: OpenShift cluster components have become healthy in approximately 60 * 15 seconds.
2026-03-14 09:14:43.824005 | 
2026-03-14 09:14:43.824171 | TASK [deploy-crc-cloud : Fail if any components are still unhealthy]
2026-03-14 09:14:43.850206 | crc | skipping: Conditional result was False
2026-03-14 09:14:43.904158 | 
2026-03-14 09:14:43.904307 | TASK [deploy-crc-cloud : Create certificate and patch secret]
2026-03-14 09:14:43.929307 | crc | skipping: Conditional result was False
2026-03-14 09:14:43.938534 | 
2026-03-14 09:14:43.938614 | TASK [deploy-crc-cloud : Wait for cluster become healthy after adding domain]
2026-03-14 09:14:43.970385 | crc | ok
2026-03-14 09:14:43.978238 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/wait_cluster_become_healthy.yaml
2026-03-14 09:14:44.011358 | 
2026-03-14 09:14:44.011455 | TASK [deploy-crc-cloud : Check if specified OpenShift components are healthy etcd|openshift-apiserver]
2026-03-14 09:14:44.504176 | crc | True
2026-03-14 09:14:44.505182 | crc | True
2026-03-14 09:14:44.564019 | crc | changed
2026-03-14 09:14:44.581033 | 
2026-03-14 09:14:44.581198 | TASK [deploy-crc-cloud : Output success message if components are healthy]
2026-03-14 09:14:44.634416 | crc | ok: OpenShift cluster components have become healthy in approximately 60 * 15 seconds.
2026-03-14 09:14:44.650264 | 
2026-03-14 09:14:44.650404 | TASK [deploy-crc-cloud : Fail if any components are still unhealthy]
2026-03-14 09:14:44.677970 | crc | skipping: Conditional result was False
2026-03-14 09:14:44.696114 | 
2026-03-14 09:14:44.696262 | TASK [deploy-crc-cloud : Patch ingress config]
2026-03-14 09:14:44.723578 | crc | skipping: Conditional result was False
2026-03-14 09:14:44.750399 | 
2026-03-14 09:14:44.750599 | TASK [deploy-crc-cloud : Patch api server]
2026-03-14 09:14:44.778860 | crc | skipping: Conditional result was False
2026-03-14 09:14:44.797203 | 
2026-03-14 09:14:44.797322 | TASK [deploy-crc-cloud : Patch default route]
2026-03-14 09:14:44.822899 | crc | skipping: Conditional result was False
2026-03-14 09:14:44.835708 | 
2026-03-14 09:14:44.835825 | TASK [deploy-crc-cloud : Wait for cluster become healthy after changing ingress api server and default route]
2026-03-14 09:14:44.870559 | crc | ok
2026-03-14 09:14:44.883514 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/wait_cluster_become_healthy.yaml
2026-03-14 09:14:44.923773 | 
2026-03-14 09:14:44.923882 | TASK [deploy-crc-cloud : Check if specified OpenShift components are healthy authentication|console|etcd|ingress|openshift-apiserver]
2026-03-14 09:14:45.455681 | crc | False
2026-03-14 09:14:45.456733 | crc | True
2026-03-14 09:14:45.456747 | crc | True
2026-03-14 09:14:45.456754 | crc | True
2026-03-14 09:14:45.456760 | crc | True
2026-03-14 09:15:47.286115 | crc | changed
2026-03-14 09:15:47.301236 | 
2026-03-14 09:15:47.301381 | TASK [deploy-crc-cloud : Output success message if components are healthy]
2026-03-14 09:15:47.344047 | crc | ok: OpenShift cluster components have become healthy in approximately 60 * 15 seconds.
2026-03-14 09:15:47.361550 | 
2026-03-14 09:15:47.361717 | TASK [deploy-crc-cloud : Fail if any components are still unhealthy]
2026-03-14 09:15:47.387126 | crc | skipping: Conditional result was False
2026-03-14 09:15:47.398669 | 
2026-03-14 09:15:47.398776 | TASK [deploy-crc-cloud : Get console route]
2026-03-14 09:15:47.423835 | crc | skipping: Conditional result was False
2026-03-14 09:15:47.443028 | 
2026-03-14 09:15:47.443109 | TASK [Start crc-cloud role when Zuul is not defined]
2026-03-14 09:15:47.467863 | crc | skipping: Conditional result was False
2026-03-14 09:15:47.480810 | 
2026-03-14 09:15:47.480896 | TASK [prepare-crc-cloud : Recreate kubelet config.json]
2026-03-14 09:15:47.505388 | crc | ok
2026-03-14 09:15:47.519885 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/inject-pull-secret.yaml
2026-03-14 09:15:47.547172 | 
2026-03-14 09:15:47.547232 | TASK [prepare-crc-cloud : Fail when openshift_pull_sec not provided]
2026-03-14 09:15:47.581773 | crc | skipping: Conditional result was False
2026-03-14 09:15:47.592972 | 
2026-03-14 09:15:47.593101 | TASK [prepare-crc-cloud : Cleanup file]
2026-03-14 09:15:48.137299 | crc | Output suppressed because no_log was given
2026-03-14 09:15:48.153585 | 
2026-03-14 09:15:48.153762 | TASK [prepare-crc-cloud : Copy pull-secret to kubelet config.json]
2026-03-14 09:15:48.555720 | crc | Output suppressed because no_log was given
2026-03-14 09:15:48.572548 | 
2026-03-14 09:15:48.572726 | TASK [prepare-crc-cloud : Replace singleqoute with doublequote]
2026-03-14 09:15:49.153619 | crc | changed: 42 replacements made
2026-03-14 09:15:49.164074 | 
2026-03-14 09:15:49.164167 | TASK [prepare-crc-cloud : Remove spaces]
2026-03-14 09:15:49.531498 | crc | changed: 20 replacements made
2026-03-14 09:15:49.547143 | 
2026-03-14 09:15:49.547276 | TASK [prepare-crc-cloud : Ensure you are logged in to the OpenShift]
2026-03-14 09:15:49.591871 | crc | ok
2026-03-14 09:15:49.609165 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/openshift_login.yaml
2026-03-14 09:15:49.641649 | 
2026-03-14 09:15:49.641756 | TASK [prepare-crc-cloud : Login to the OpenShift]
2026-03-14 09:15:50.067432 | crc | WARNING: Using insecure TLS client config. Setting this option is not supported!
2026-03-14 09:15:50.202841 | crc |
2026-03-14 09:15:50.202923 | crc | Login successful.
2026-03-14 09:15:50.226655 | crc |
2026-03-14 09:15:50.226706 | crc | You have access to 65 projects, the list has been suppressed. You can list all projects with 'oc projects'
2026-03-14 09:15:50.253871 | crc |
2026-03-14 09:15:50.253923 | crc | Using project "default".
2026-03-14 09:15:50.685663 | crc | ok
2026-03-14 09:15:50.702187 | 
2026-03-14 09:15:50.702331 | TASK [prepare-crc-cloud : Post tasks after crc-cloud script execution]
2026-03-14 09:15:50.746405 | crc | ok
2026-03-14 09:15:50.774459 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/post.yaml
2026-03-14 09:15:50.814582 | 
2026-03-14 09:15:50.814722 | LOOP [prepare-crc-cloud : Set DNS nameservers provided by cloud provider]
2026-03-14 09:15:51.148511 | crc | changed: line added
2026-03-14 09:15:51.438465 | crc | changed: line added
2026-03-14 09:15:51.465103 | 
2026-03-14 09:15:51.465330 | TASK [prepare-crc-cloud : Increase dnsmasq cache size]
2026-03-14 09:15:51.814671 | crc | changed: line added
2026-03-14 09:15:51.832302 | 
2026-03-14 09:15:51.832445 | TASK [prepare-crc-cloud : Reload NetworkManager after adding cache and DNS server]
2026-03-14 09:15:52.798405 | crc | changed
2026-03-14 09:15:52.821432 | 
2026-03-14 09:15:52.821590 | TASK [prepare-crc-cloud : Ensure kube config does not exists]
2026-03-14 09:15:53.211331 | crc | ok
2026-03-14 09:15:53.229414 | 
2026-03-14 09:15:53.229574 | TASK [prepare-crc-cloud : Copy new generated KUBECONFIG]
2026-03-14 09:15:53.257574 | crc | skipping: Conditional result was False
2026-03-14 09:15:53.278080 | 
2026-03-14 09:15:53.278281 | TASK [prepare-crc-cloud : Delete all openshift-marketplace pods]
2026-03-14 09:15:53.796265 | crc | pod "certified-operators-mss9d" deleted
2026-03-14 09:15:53.808260 | crc | pod "community-operators-kzj4r" deleted
2026-03-14 09:15:53.815979 | crc | pod "marketplace-operator-79b997595-xtq2f" deleted
2026-03-14 09:15:53.838898 | crc | pod "redhat-marketplace-mmpq6" deleted
2026-03-14 09:15:53.847329 | crc | pod "redhat-operators-jfzjg" deleted
2026-03-14 09:15:57.380962 | crc | ok
2026-03-14 09:15:57.398511 | 
2026-03-14 09:15:57.398654 | TASK [prepare-crc-cloud : Create workaround for expired cert]
2026-03-14 09:15:57.483319 | crc | ok
2026-03-14 09:15:57.506540 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/cluster_cert.yaml
2026-03-14 09:15:57.586469 | 
2026-03-14 09:15:57.586596 | LOOP [prepare-crc-cloud : Re-run accept certificates in Pending state few times]
2026-03-14 09:15:58.271467 | crc | ok
2026-03-14 09:15:58.271889 | crc | changed: All items complete
2026-03-14 09:15:58.271948 | 
2026-03-14 09:16:10.778177 | crc | ok
2026-03-14 09:16:21.213167 | crc | ok
2026-03-14 09:16:21.234486 | 
2026-03-14 09:16:21.234713 | TASK [prepare-crc-cloud : Create service account for - auto-csr-approver]
2026-03-14 09:16:21.991004 | crc | ok
2026-03-14 09:16:22.007358 | 
2026-03-14 09:16:22.007487 | TASK [prepare-crc-cloud : Apply cluster cert approver]
2026-03-14 09:16:22.641027 | crc | serviceaccount/csr-approver-sa unchanged
2026-03-14 09:16:22.666003 | crc | clusterrole.rbac.authorization.k8s.io/csr-approver unchanged
2026-03-14 09:16:22.689528 | crc | clusterrolebinding.rbac.authorization.k8s.io/csr-approver-binding unchanged
2026-03-14 09:16:22.729127 | crc | cronjob.batch/auto-csr-approver unchanged
2026-03-14 09:16:23.064958 | crc | ok
2026-03-14 09:16:23.081188 | 
2026-03-14 09:16:23.081324 | TASK [prepare-crc-cloud : Pause machineconfigpool]
2026-03-14 09:16:23.115463 | crc | ok
2026-03-14 09:16:23.137008 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/pause_mcp.yaml
2026-03-14 09:16:23.182195 | 
2026-03-14 09:16:23.182339 | LOOP [prepare-crc-cloud : Pause the machineconfig configuration if mcp is not needed]
2026-03-14 09:16:23.765507 | crc | ok
2026-03-14 09:16:24.205223 | crc | ok
2026-03-14 09:16:24.223644 | 
2026-03-14 09:16:24.223815 | TASK [prepare-crc-cloud : Print current machineconfigpool]
2026-03-14 09:16:24.697581 | crc | NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
2026-03-14 09:16:24.697712 | crc | master   rendered-master-f83a149c7490c3b0bf850df839837383   False     True       True       1              0                   0                     1                      384d
2026-03-14 09:16:24.710598 | crc | worker   rendered-worker-594cc07c9feefb5e37f2530ac457d36c   True      False      False      0              0                   0                     0                      384d
2026-03-14 09:16:24.782294 | crc | ok
2026-03-14 09:16:24.799341 | 
2026-03-14 09:16:24.799476 | TASK [prepare-crc-cloud : Set low profile to etcd to avoid flakiness]
2026-03-14 09:16:24.954956 | crc | skipping: Conditional result was False
2026-03-14 09:16:24.968668 | 
2026-03-14 09:16:24.968895 | TASK [prepare-crc-cloud : Fix machineconfigpool degradation]
2026-03-14 09:16:25.005875 | crc | skipping: Conditional result was False
2026-03-14 09:16:25.023183 | 
2026-03-14 09:16:25.023315 | TASK [prepare-crc-cloud : Remove drain machineconfig for the node]
2026-03-14 09:16:25.051804 | crc | skipping: Conditional result was False
2026-03-14 09:16:25.071000 | 
2026-03-14 09:16:25.071144 | TASK [prepare-crc-cloud : Post machineconfigpool tasks]
2026-03-14 09:16:25.098542 | crc | skipping: Conditional result was False
2026-03-14 09:16:25.122030 | 
2026-03-14 09:16:25.122189 | TASK [prepare-crc-cloud : Enable monitoring and cluster-monitoring-operator]
2026-03-14 09:16:25.151733 | crc | skipping: Conditional result was False
2026-03-14 09:16:25.181938 | 
2026-03-14 09:16:25.182119 | TASK [Run crc-extracted preparation]
2026-03-14 09:16:25.209308 | crc | skipping: Conditional result was False
2026-03-14 09:16:25.220773 | 
2026-03-14 09:16:25.220893 | TASK [Fail when image build date does not exists]
2026-03-14 09:16:25.246425 | crc | skipping: Conditional result was False
2026-03-14 09:16:25.288886 | 
2026-03-14 09:16:25.289011 | PLAY [Set IOPS and Read/Write disk limitation]
2026-03-14 09:16:25.324696 | 
2026-03-14 09:16:25.324812 | TASK [Set limit for disk usage - IOPS and Read/Write]
2026-03-14 09:16:25.408021 | controller | ok
2026-03-14 09:16:25.421648 | crc | ok
2026-03-14 09:16:25.463780 | 
2026-03-14 09:16:25.463910 | TASK [disk-usage-limitation : Get block device major:minor]
2026-03-14 09:16:25.823537 | crc | 252:0
2026-03-14 09:16:25.903312 | controller | 252:0
2026-03-14 09:16:26.042895 | crc | ok
2026-03-14 09:16:26.056854 | controller | ok
2026-03-14 09:16:26.069368 | 
2026-03-14 09:16:26.069500 | LOOP [disk-usage-limitation : Ensure that some cgroup dirs exists]
2026-03-14 09:16:26.329661 | controller | ok: "/sys/fs/cgroup/init.scope"
2026-03-14 09:16:26.330102 | controller | ok: All items complete
2026-03-14 09:16:26.330160 | 
2026-03-14 09:16:26.419600 | crc | ok: "/sys/fs/cgroup/init.scope"
2026-03-14 09:16:26.533612 | controller | changed: "/sys/fs/cgroup/machine.slice"
2026-03-14 09:16:26.701963 | crc | ok: "/sys/fs/cgroup/machine.slice"
2026-03-14 09:16:26.734440 | controller | ok: "/sys/fs/cgroup/system.slice"
2026-03-14 09:16:26.925083 | controller | ok: "/sys/fs/cgroup/user.slice"
2026-03-14 09:16:26.994084 | crc | ok: "/sys/fs/cgroup/system.slice"
2026-03-14 09:16:27.282496 | crc | ok: "/sys/fs/cgroup/user.slice"
2026-03-14 09:16:27.299485 | 
2026-03-14 09:16:27.299621 | TASK [disk-usage-limitation : Ensure systemd configuration directory]
2026-03-14 09:16:27.549894 | controller | changed
2026-03-14 09:16:27.709609 | crc | ok
2026-03-14 09:16:27.724338 | 
2026-03-14 09:16:27.724512 | TASK [disk-usage-limitation : Enable IO limit via override configuration]
2026-03-14 09:16:28.351780 | controller | changed
2026-03-14 09:16:28.611655 | crc | changed
2026-03-14 09:16:28.624470 | 
2026-03-14 09:16:28.624621 | TASK [disk-usage-limitation : Flush handlers that will call systemd daemon-reload]
2026-03-14 09:16:28.648603 | 
2026-03-14 09:16:28.648742 | TASK [disk-usage-limitation : Flush handlers that will call systemd daemon-reload]
2026-03-14 09:16:28.665122 | 
2026-03-14 09:16:28.665206 | TASK [disk-usage-limitation : Reload systemd daemon after enabling IOAccounting]
2026-03-14 09:16:29.500436 | controller | ok
2026-03-14 09:16:30.622771 | crc | ok
2026-03-14 09:16:30.635612 | 
2026-03-14 09:16:30.635787 | TASK [disk-usage-limitation : Wait for system.slice cgroup to have io.max file]
2026-03-14 09:16:30.897410 | controller | ok
2026-03-14 09:16:31.004659 | crc | ok
2026-03-14 09:16:31.019397 | 
2026-03-14 09:16:31.019543 | LOOP [disk-usage-limitation : Set IOPS limit in the root cgroup]
2026-03-14 09:16:31.304040 | controller | ok
2026-03-14 09:16:31.306456 | controller | ok: All items complete
2026-03-14 09:16:31.306551 | 
2026-03-14 09:16:31.409214 | crc | ok
2026-03-14 09:16:31.523655 | controller | ok
2026-03-14 09:16:31.681548 | crc | ok
2026-03-14 09:16:31.733588 | controller | ok
2026-03-14 09:16:31.938652 | controller | ok
2026-03-14 09:16:32.006800 | crc | ok
2026-03-14 09:16:32.281831 | crc | ok
2026-03-14 09:16:32.301307 | 
2026-03-14 09:16:32.301514 | TASK [disk-usage-limitation : Verify the IOPS limit]
2026-03-14 09:16:32.714318 | crc | init
2026-03-14 09:16:32.716389 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:16:32.716713 | crc | machine
2026-03-14 09:16:32.718606 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:16:32.718827 | crc | system
2026-03-14 09:16:32.719999 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:16:32.720173 | crc | user
2026-03-14 09:16:32.721982 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:16:32.607951 | controller | init
2026-03-14 09:16:32.607993 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:16:32.608001 | controller | machine
2026-03-14 09:16:32.608006 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:16:32.608010 | controller | system
2026-03-14 09:16:32.608015 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:16:32.608019 | controller | user
2026-03-14 09:16:32.608024 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:16:32.900467 | controller | ok
2026-03-14 09:16:32.917627 | crc | ok
2026-03-14 09:16:32.931265 | 
2026-03-14 09:16:32.931409 | TASK [disk-usage-limitation : Check if kubepods.slice exists]
2026-03-14 09:16:33.198451 | controller | ok
2026-03-14 09:16:33.304409 | crc | ok
2026-03-14 09:16:33.313046 | 
2026-03-14 09:16:33.313130 | TASK [disk-usage-limitation : Set the limit]
2026-03-14 09:16:33.878208 | controller | skipping: Conditional result was False
2026-03-14 09:16:33.895235 | crc | ok
2026-03-14 09:16:33.908940 | 
2026-03-14 09:16:33.909073 | TASK [disk-usage-limitation : Verify the IOPS limit - kubepods slice]
2026-03-14 09:16:34.289824 | crc | kubepods
2026-03-14 09:16:34.291846 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:16:34.513102 | controller | skipping: Conditional result was False
2026-03-14 09:16:34.531571 | crc | ok
2026-03-14 09:16:34.546094 | 
2026-03-14 09:16:34.546275 | TASK [disk-usage-limitation : Print the current io.max value]
2026-03-14 09:16:34.595224 | controller | ok:
2026-03-14 09:16:34.595513 | controller | init
2026-03-14 09:16:34.595553 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:16:34.595581 | controller | machine
2026-03-14 09:16:34.595608 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:16:34.595667 | controller | system
2026-03-14 09:16:34.595717 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:16:34.595743 | controller | user
2026-03-14 09:16:34.595765 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:16:34.635173 | crc | ok:
2026-03-14 09:16:34.635252 | crc | init
2026-03-14 09:16:34.635285 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:16:34.635311 | crc | machine
2026-03-14 09:16:34.635335 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:16:34.635359 | crc | system
2026-03-14 09:16:34.635382 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:16:34.635406 | crc | user
2026-03-14 09:16:34.635428 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 kubepods
2026-03-14 09:16:34.635451 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:16:34.704750 | 
2026-03-14 09:16:34.704837 | PLAY RECAP
2026-03-14 09:16:34.704879 | controller | ok: 10 changed: 3 unreachable: 0 failed: 0 skipped: 3 rescued: 0 ignored: 0
2026-03-14 09:16:34.704905 | crc | ok: 112 changed: 37 unreachable: 0 failed: 0 skipped: 28 rescued: 0 ignored: 0
2026-03-14 09:16:34.704922 | 
2026-03-14 09:16:34.832491 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/prepare-crc.yaml@master]
2026-03-14 09:16:34.841928 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/podman.yaml@master]
2026-03-14 09:16:35.496104 | 
2026-03-14 09:16:35.496213 | PLAY [all,!compute]
2026-03-14 09:16:35.532732 | 
2026-03-14 09:16:35.532858 | TASK [Enable container-tools rhel-modules for c8/crc node]
2026-03-14 09:16:36.096570 | crc | skipping: Conditional result was False
2026-03-14 09:16:36.097619 | controller | skipping: Conditional result was False
2026-03-14 09:16:36.111560 | 
2026-03-14 09:16:36.111766 | TASK [Install podman]
2026-03-14 09:16:36.205739 | crc | skipping: Conditional result was False
2026-03-14 09:17:22.249396 | controller | changed
2026-03-14 09:17:22.262373 | 
2026-03-14 09:17:22.262502 | TASK [Set var name for quay login zuul secret]
2026-03-14 09:17:22.342912 | controller | skipping: Conditional result was False
2026-03-14 09:17:22.354197 | crc | skipping: Conditional result was False
2026-03-14 09:17:22.369917 | 
2026-03-14 09:17:22.370064 | TASK [Print the username]
2026-03-14 09:17:22.831363 | controller | openstack-k8s-operators+cirobot
2026-03-14 09:17:22.966277 | crc | skipping: Conditional result was False
2026-03-14 09:17:22.979331 | controller | ok: Runtime: 0:00:00.004427
2026-03-14 09:17:22.991982 | 
2026-03-14 09:17:22.992158 | TASK [Fail when user or password is not set]
2026-03-14 09:17:23.069670 | controller | skipping: Conditional result was False
2026-03-14 09:17:23.090809 | crc | skipping: Conditional result was False
2026-03-14 09:17:23.100333 | 
2026-03-14 09:17:23.100444 | TASK [Set vars for quay login]
2026-03-14 09:17:23.174009 | controller | Output suppressed because no_log was given
2026-03-14 09:17:23.193978 | 
2026-03-14 09:17:23.194112 | LOOP [Perform container registry login(s) with podman]
2026-03-14 09:17:23.949286 | controller | Output suppressed because no_log was given
2026-03-14 09:17:23.992922 | 
2026-03-14 09:17:23.993519 | LOOP [Perform container registry login(s) with buildah]
2026-03-14 09:17:24.075991 | controller | Output suppressed because no_log was given
2026-03-14 09:17:24.076282 | 
2026-03-14 09:17:24.088945 | 
2026-03-14 09:17:24.089056 | TASK [Set Insecure registry for content provider]
2026-03-14 09:17:24.145264 | crc | skipping: Conditional result was False
2026-03-14 09:17:24.503820 | controller | changed: Block inserted
2026-03-14 09:17:24.579019 | 
2026-03-14 09:17:24.579136 | PLAY RECAP
2026-03-14 09:17:24.579182 | controller | ok: 5 changed: 4 unreachable: 0 failed: 0 skipped: 4 rescued: 0 ignored: 0
2026-03-14 09:17:24.579208 | crc | ok: 0 changed: 0 unreachable: 0 failed: 0 skipped: 9 rescued: 0 ignored: 0
2026-03-14 09:17:24.579226 | 
2026-03-14 09:17:24.677483 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/podman.yaml@master]
2026-03-14 09:17:24.687527 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/data_plane_adoption/dpa_RH_subscription_registry_login.yaml@master]
2026-03-14 09:17:25.320198 | 
2026-03-14 09:17:25.320314 | PLAY [rh-subscription]
2026-03-14 09:17:25.320410 | 
2026-03-14 09:17:25.320448 | PLAY RECAP
2026-03-14 09:17:25.320476 | 
2026-03-14 09:17:25.426443 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/data_plane_adoption/dpa_RH_subscription_registry_login.yaml@master]
2026-03-14 09:17:25.430293 | PRE-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/multinode-customizations.yml@main]
2026-03-14 09:17:26.037179 | 
2026-03-14 09:17:26.037289 | PLAY [Tweak CRC node]
2026-03-14 09:17:26.067542 | 
2026-03-14 09:17:26.067670 | TASK [Load network parameters]
2026-03-14 09:17:26.791175 | crc | ok
2026-03-14 09:17:26.805565 | 
2026-03-14 09:17:26.805753 | TASK [Check which dnsmasq config we must edit]
2026-03-14 09:17:27.235170 | crc | ok
2026-03-14 09:17:27.249850 | 
2026-03-14 09:17:27.249985 | TASK [Ensure crc knows about its second NIC]
2026-03-14 09:17:28.131309 | crc | changed
2026-03-14 09:17:28.146307 | 
2026-03-14 09:17:28.146434 | TASK [Ensure crc does not get "public" DNS]
2026-03-14 09:17:28.947011 | crc | changed
2026-03-14 09:17:28.963486 | 
2026-03-14 09:17:28.963623 | TASK [Restart NetworkManager]
2026-03-14 09:17:30.109733 | crc | changed
2026-03-14 09:17:30.129370 | 
2026-03-14 09:17:30.129515 | LOOP [Configure dns forwarders]
2026-03-14 09:17:30.604071 | crc | ok
2026-03-14 09:17:30.950157 | crc | ok
2026-03-14 09:17:30.968613 | 
2026-03-14 09:17:30.968964 | TASK [Configure local DNS for CRC pod]
2026-03-14 09:17:31.430937 | crc | ok
2026-03-14 09:17:31.446417 | 
2026-03-14 09:17:31.446555 | TASK [Configure dnsmasq listen-address to listen on both br-ex and ci-private-network]
2026-03-14 09:17:31.939094 | crc | changed: line added
2026-03-14 09:17:31.954831 | 
2026-03-14 09:17:31.954997 | TASK [Restart native dnsmasq service]
2026-03-14 09:17:32.549664 | crc | changed
2026-03-14 09:17:32.569756 | 
2026-03-14 09:17:32.569933 | TASK [Stop dnsmasq]
2026-03-14 09:17:32.597764 | crc | skipping: Conditional result was False
2026-03-14 09:17:32.615876 | 
2026-03-14 09:17:32.616052 | TASK [Make sure that crc-dnsmasq is not running]
2026-03-14 09:17:32.643432 | crc | skipping: Conditional result was False
2026-03-14 09:17:32.659708 | 
2026-03-14 09:17:32.659857 | TASK [Start dnsmasq]
2026-03-14 09:17:32.686810 | crc | skipping: Conditional result was False
2026-03-14 09:17:32.706115 | 
2026-03-14 09:17:32.706250 | TASK [Wait for CRC to be ready]
2026-03-14 09:17:33.279557 | crc | WARNING: Using insecure TLS client config. Setting this option is not supported!
2026-03-14 09:17:33.418241 | crc |
2026-03-14 09:17:33.418391 | crc | Login successful.
2026-03-14 09:17:33.436495 | crc |
2026-03-14 09:17:33.436571 | crc | You have access to 65 projects, the list has been suppressed. You can list all projects with 'oc projects'
2026-03-14 09:17:33.460833 | crc |
2026-03-14 09:17:33.460885 | crc | Using project "default".
2026-03-14 09:17:33.761029 | crc | changed
2026-03-14 09:17:33.768877 | 
2026-03-14 09:17:33.768943 | TASK [Set insecure registry on crc node]
2026-03-14 09:17:33.818471 | crc | ok
2026-03-14 09:17:33.845490 | crc | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/ci/playbooks/tasks/set_crc_insecure_registry.yml
2026-03-14 09:17:33.859391 | 
2026-03-14 09:17:33.859485 | TASK [Patch the image.config.openshift.io resource to include insecure registry]
2026-03-14 09:17:34.418584 | crc | image.config.openshift.io/cluster patched
2026-03-14 09:17:34.909654 | crc | changed
2026-03-14 09:17:34.926587 | 
2026-03-14 09:17:34.926760 | TASK [Patch the image.config.openshift.io resource to allow registries]
2026-03-14 09:17:35.483162 | crc | image.config.openshift.io/cluster patched
2026-03-14 09:17:35.540193 | crc | changed
2026-03-14 09:17:35.555454 | 
2026-03-14 09:17:35.555593 | LOOP [Add additional allowed registries]
2026-03-14 09:17:35.611901 | 
2026-03-14 09:17:35.612182 | TASK [Ensure registries.conf.d exists]
2026-03-14 09:17:36.115490 | crc | ok
2026-03-14 09:17:36.130911 | 
2026-03-14 09:17:36.131051 | TASK [Set Insecure registry for content provider]
2026-03-14 09:17:36.569045 | crc | changed: File created
2026-03-14 09:17:36.580270 | 
2026-03-14 09:17:36.580417 | TASK [Set registry mirror override]
2026-03-14 09:17:36.616797 | crc | skipping: Conditional result was False
2026-03-14 09:17:36.635017 | 
2026-03-14 09:17:36.635231 | TASK [Restart crio]
2026-03-14 09:17:38.493419 | crc | changed
2026-03-14 09:17:38.540776 | 
2026-03-14 09:17:38.540909 | PLAY [Tweak Controller]
2026-03-14 09:17:38.570348 | 
2026-03-14 09:17:38.570491 | TASK [Install other packages]
2026-03-14 09:17:42.380932 | controller | changed
2026-03-14 09:17:42.393183 | 
2026-03-14 09:17:42.393312 | LOOP [Ensure we know compute host keys]
2026-03-14 09:17:42.783091 | controller | changed
2026-03-14 09:17:42.783505 | 
2026-03-14 09:17:43.021418 | controller | changed
2026-03-14 09:17:43.039035 | 
2026-03-14 09:17:43.039149 | TASK [Generate an ssh keypair]
2026-03-14 09:17:43.283732 | controller | Generating public/private ecdsa key pair.
2026-03-14 09:17:43.283873 | controller | Your identification has been saved in /home/zuul/.ssh/id_cifw
2026-03-14 09:17:43.283925 | controller | Your public key has been saved in /home/zuul/.ssh/id_cifw.pub
2026-03-14 09:17:43.283931 | controller | The key fingerprint is:
2026-03-14 09:17:43.283936 | controller | SHA256:LIposkjURW+shf4rHtH9JSEa1WV5iKRN+kZQ8dv/hgg zuul@np0005647135.novalocal
2026-03-14 09:17:43.283940 | controller | The key's randomart image is:
2026-03-14 09:17:43.283945 | controller | +---[ECDSA 256]---+
2026-03-14 09:17:43.283949 | controller | |     .   o+=+oo  |
2026-03-14 09:17:43.283953 | controller | |    . + . *ooo . |
2026-03-14 09:17:43.283958 | controller | |     o * + + ..  |
2026-03-14 09:17:43.283962 | controller | |  . o =.+ + . o  |
2026-03-14 09:17:43.283966 | controller | | . . +.oS. + o . |
2026-03-14 09:17:43.283971 | controller | |.. . .o.  E o   .|
2026-03-14 09:17:43.283975 | controller | |oo. .. .   o . ..|
2026-03-14 09:17:43.283979 | controller | |=.   .. .   . . o|
2026-03-14 09:17:43.283983 | controller | |o   ....       ..|
2026-03-14 09:17:43.283988 | controller | +----[SHA256]-----+
2026-03-14 09:17:43.575154 | controller | changed
2026-03-14 09:17:43.589442 | 
2026-03-14 09:17:43.589586 | TASK [Get public key]
2026-03-14 09:17:43.833610 | controller | ok
2026-03-14 09:17:43.843823 | 
2026-03-14 09:17:43.843919 | TASK [Register pub key as a fact]
2026-03-14 09:17:43.909857 | controller | ok
2026-03-14 09:17:43.918628 | 
2026-03-14 09:17:43.918731 | TASK [Load network parameters]
2026-03-14 09:17:44.170768 | controller | ok
2026-03-14 09:17:44.183554 | 
2026-03-14 09:17:44.183710 | TASK [Check if cloud-init is overriding NM settings]
2026-03-14 09:17:44.398763 | controller | ok
2026-03-14 09:17:44.411537 | 
2026-03-14 09:17:44.411659 | TASK [Remove cloud-init DNS override if present]
2026-03-14 09:17:44.439300 | controller | skipping: Conditional result was False
2026-03-14 09:17:44.455215 | 
2026-03-14 09:17:44.455344 | TASK [Reload the NetworkManager to pick the changes]
2026-03-14 09:17:44.543990 | controller | skipping: Conditional result was False
2026-03-14 09:17:44.600161 | 
2026-03-14 09:17:44.600300 | TASK [Get the default iface connection]
2026-03-14 09:17:44.881554 | controller | System eth0
2026-03-14 09:17:45.139642 | controller | changed
2026-03-14 09:17:45.153846 | 
2026-03-14 09:17:45.154909 | TASK [Prepend CRC DNS server in the controllers default Network Manager connection configuation]
2026-03-14 09:17:45.689380 | controller | Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/5)
2026-03-14 09:17:45.725654 | controller | changed
2026-03-14 09:17:45.747858 | 
2026-03-14 09:17:45.747943 | PLAY [Ensure persistent SSH access]
2026-03-14 09:17:45.777190 | 
2026-03-14 09:17:45.777266 | TASK [Inject key in ansible_user]
2026-03-14 09:17:46.218819 | controller | changed
2026-03-14 09:17:46.480623 | crc | changed
2026-03-14 09:17:46.494341 | 
2026-03-14 09:17:46.494508 | TASK [Inject key in root]
2026-03-14 09:17:46.824521 | controller | changed
2026-03-14 09:17:46.964631 | crc | changed
2026-03-14 09:17:47.048071 | 
2026-03-14 09:17:47.048176 | PLAY [Add cloud-admin user on Compute]
2026-03-14 09:17:47.049336 | 
2026-03-14 09:17:47.049384 | PLAY [Set some fancy hostname]
2026-03-14 09:17:47.067074 | 
2026-03-14 09:17:47.067143 | TASK [Inject hostname in configuration file]
2026-03-14 09:17:47.785112 | controller | changed
2026-03-14 09:17:47.828135 | 
2026-03-14 09:17:47.828340 | PLAY RECAP
2026-03-14 09:17:47.828406 | controller | ok: 12 changed: 8 unreachable: 0 failed: 0 skipped: 2 rescued: 0 ignored: 0
2026-03-14 09:17:47.828442 | crc | ok: 18 changed: 12 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0
2026-03-14 09:17:47.828467 | 
2026-03-14 09:17:47.945910 | PRE-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/multinode-customizations.yml@main]
2026-03-14 09:17:47.950069 | PRE-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/e2e-prepare.yml@main]
2026-03-14 09:17:48.603865 | 
2026-03-14 09:17:48.603966 | PLAY [Run ci/playbooks/e2e-prepare.yml]
2026-03-14 09:17:48.626564 | 
2026-03-14 09:17:48.626696 | TASK [Clone repos in the job workspace]
2026-03-14 09:17:48.657273 | controller | ok
2026-03-14 09:17:48.675365 | 
2026-03-14 09:17:48.675467 | TASK [prepare-workspace : Start zuul_console daemon.]
2026-03-14 09:17:49.180609 | controller | ok
2026-03-14 09:17:49.196069 | 
2026-03-14 09:17:49.196225 | TASK [prepare-workspace : Synchronize src repos to workspace directory.]
2026-03-14 09:18:01.688938 | controller | Output suppressed because no_log was given
2026-03-14 09:18:01.715394 | 
2026-03-14 09:18:01.715563 | TASK [Create zuul-output directory]
2026-03-14 09:18:02.124029 | controller | ok
2026-03-14 09:18:02.138098 | 
2026-03-14 09:18:02.138238 | TASK [Install required packages]
2026-03-14 09:18:03.186035 | controller | ok: Nothing to do
2026-03-14 09:18:03.203717 | 
2026-03-14 09:18:03.203900 | TASK [Output pip related things]
2026-03-14 09:18:03.840012 | controller | pip 21.3.1 from /usr/lib/python3.9/site-packages/pip (python 3.9)
2026-03-14 09:18:04.255557 | controller | changed
2026-03-14 09:18:04.270878 | 
2026-03-14 09:18:04.271160 | TASK [Install requirements]
2026-03-14 09:19:01.647262 | controller | changed
2026-03-14 09:19:01.660653 | 
2026-03-14 09:19:01.660751 | TASK [Add cronjob to trigger job stats]
2026-03-14 09:19:01.726055 | controller | skipping: Conditional result was False
2026-03-14 09:19:01.733556 | 
2026-03-14 09:19:01.733644 | LOOP [Construct project change list]
2026-03-14 09:19:01.803667 | controller | ok:
2026-03-14 09:19:01.803943 | controller | {
2026-03-14 09:19:01.803979 | controller |   "branch": "main",
2026-03-14 09:19:01.804006 | controller |   "change": "352",
2026-03-14 09:19:01.804030 | controller |   "change_message": "openstack-k8s-operators dependency bump branch: main\n\nAutomated changes by [create-pull-request](https://github.com/peter-evans/create-pull-request) GitHub action",
2026-03-14 09:19:01.804061 | controller |   "change_url": "https://github.com/openstack-k8s-operators/watcher-operator/pull/352",
2026-03-14 09:19:01.804085 | controller |   "commit_id": "0d9467a022a6d6a8b6465cdbc9e4aefc533e796a",
2026-03-14 09:19:01.804108 | controller |   "patchset": "0d9467a022a6d6a8b6465cdbc9e4aefc533e796a",
2026-03-14 09:19:01.804130 | controller |   "project": {
2026-03-14 09:19:01.804152 | controller |     "canonical_hostname": "github.com",
2026-03-14 09:19:01.804174 | controller |     "canonical_name": "github.com/openstack-k8s-operators/watcher-operator",
2026-03-14 09:19:01.804196 | controller |     "name": "openstack-k8s-operators/watcher-operator",
2026-03-14 09:19:01.804218 | controller |     "short_name": "watcher-operator",
2026-03-14 09:19:01.804239 | controller |     "src_dir": "src/github.com/openstack-k8s-operators/watcher-operator"
2026-03-14 09:19:01.804262 | controller |   },
2026-03-14 09:19:01.804284 | controller |   "topic": null
2026-03-14 09:19:01.804305 | controller | }
2026-03-14 09:19:01.856748 | 
2026-03-14 09:19:01.856846 | PLAY [Run ci/playbooks/build_runner_image.yml]
2026-03-14 09:19:01.889151 | 
2026-03-14 09:19:01.889277 | TASK [Filter out host if needed]
2026-03-14 09:19:01.901116 | controller | skipping: end_host conditional evaluated to False, continuing execution for controller
2026-03-14 09:19:01.907016 | 
2026-03-14 09:19:01.907069 | TASK [Filter out host if needed]
2026-03-14 09:19:01.915483 | crc | skipping: end_host conditional evaluated to False, continuing execution for crc
2026-03-14 09:19:01.921892 | 
2026-03-14 09:19:01.921948 | TASK [Get git tag for image tagging]
2026-03-14 09:19:02.525345 | controller | skipping: Conditional result was False
2026-03-14 09:19:02.526388 | crc | skipping: Conditional result was False
2026-03-14 09:19:02.540398 | 
2026-03-14 09:19:02.540575 | TASK [Set openstack-runner image]
2026-03-14 09:19:02.573740 | controller | skipping: Conditional result was False
2026-03-14 09:19:02.596135 | crc | skipping: Conditional result was False
2026-03-14 09:19:02.609625 | 
2026-03-14 09:19:02.609825 | TASK [Build openstack-runner image]
2026-03-14 09:19:02.647522 | controller | skipping: Conditional result was False
2026-03-14 09:19:02.668845 | crc | skipping: Conditional result was False
2026-03-14 09:19:02.680336 | 
2026-03-14 09:19:02.680433 | TASK [Push openstack-runner image]
2026-03-14 09:19:02.711947 | controller | skipping: Conditional result was False
2026-03-14 09:19:02.723001 | crc | skipping: Conditional result was False
2026-03-14 09:19:02.732296 | 
2026-03-14 09:19:02.732389 | TASK [Make sure ci-framework directory exists]
2026-03-14 09:19:02.766232 | controller | skipping: Conditional result was False
2026-03-14 09:19:02.777264 | crc | skipping: Conditional result was False
2026-03-14 09:19:02.789078 | 
2026-03-14 09:19:02.789182 | TASK [Create EDPM ansible var file]
2026-03-14 09:19:02.821115 | controller | skipping: Conditional result was False
2026-03-14 09:19:02.832772 | crc | skipping: Conditional result was False
2026-03-14 09:19:02.915967 | 
2026-03-14 09:19:02.916094 | PLAY RECAP
2026-03-14 09:19:02.916152 | controller | ok: 7 changed: 3 unreachable: 0 failed: 0 skipped: 7 rescued: 0 ignored: 0
2026-03-14 09:19:02.916197 | crc | ok: 0 changed: 0 unreachable: 0 failed: 0 skipped: 6 rescued: 0 ignored: 0
2026-03-14 09:19:02.916223 | 
2026-03-14 09:19:03.042143 | PRE-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/e2e-prepare.yml@main]
2026-03-14 09:19:03.053853 | PRE-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/dump_zuul_data.yml@main]
2026-03-14 09:19:03.679977 | 
2026-03-14 09:19:03.680085 | PLAY [Run ci/playbooks/dump_zuul_data.yml]
2026-03-14 09:19:03.703920 | 
2026-03-14 09:19:03.704015 | TASK [Create zuul-output directory]
2026-03-14 09:19:04.221108 | controller | changed
2026-03-14 09:19:04.234945 | 
2026-03-14 09:19:04.235114 | TASK [Slurp Zuul inventory test]
2026-03-14 09:19:04.659643 | controller -> localhost | ok
2026-03-14 09:19:04.674139 | 
2026-03-14 09:19:04.674268 | TASK [Save zuul inventory]
2026-03-14 09:19:05.589102 | controller | changed
2026-03-14 09:19:05.601469 | 
2026-03-14 09:19:05.601603 | TASK [Save zuul vars without the change_message]
2026-03-14 09:19:06.324595 | controller | changed
2026-03-14 09:19:06.368900 | 
2026-03-14 09:19:06.368978 | PLAY RECAP
2026-03-14 09:19:06.369035 | controller | ok: 4 changed: 3 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0
2026-03-14 09:19:06.369063 | 
2026-03-14 09:19:06.465907 | PRE-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/dump_zuul_data.yml@main]
2026-03-14 09:19:06.471556 | PRE-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/kuttl/kuttl-from-operator-deps.yaml@main]
2026-03-14 09:19:07.035412 | 
2026-03-14 09:19:07.035532 | PLAY [Run ci/playbooks/kuttl/deploy-deps.yml]
2026-03-14 09:19:07.059552 | 
2026-03-14 09:19:07.059694 | TASK [Run kuttl deploy-deps playbook]
2026-03-14 09:19:08.323923 | controller |
2026-03-14 09:19:09.488012 | controller | PLAY [Bootstrap playbook] ******************************************************
2026-03-14 09:19:09.488145 | controller |
2026-03-14 09:19:09.488166 | controller | TASK [Gathering Facts ] ********************************************************
2026-03-14 09:19:09.488182 | controller | Saturday 14 March 2026  09:19:08 +0000 (0:00:00.053)       0:00:00.053 ********
2026-03-14 09:19:09.488197 | controller | Saturday 14 March 2026  09:19:08 +0000 (0:00:00.051)       0:00:00.051 ********
2026-03-14 09:19:09.488227 | controller | ok: [controller]
2026-03-14 09:19:09.532848 | controller |
2026-03-14 09:19:09.532918 | controller | TASK [cifmw_setup : Set custom cifmw PATH reusable fact cifmw_path={{ ansible_user_dir }}/.crc/bin:{{ ansible_user_dir }}/.crc/bin/oc:{{ ansible_user_dir }}/bin:{{ ansible_env.PATH }}, cacheable=True] ***
2026-03-14 09:19:09.532935 | controller | Saturday 14 March 2026  09:19:09 +0000 (0:00:01.164)       0:00:01.217 ********
2026-03-14 09:19:09.532946 | controller | Saturday 14 March 2026  09:19:09 +0000 (0:00:01.164)       0:00:01.216 ********
2026-03-14 09:19:09.532965 | controller | ok: [controller]
2026-03-14 09:19:09.638119 | controller |
2026-03-14 09:19:09.638170 | controller | TASK [cifmw_setup : Get customized parameters ci_framework_params={{
2026-03-14 09:19:09.638177 | controller |   hostvars[inventory_hostname] |
2026-03-14 09:19:09.638185 | controller |   dict2items |
2026-03-14 09:19:09.638190 | controller |   selectattr("key", "match",
2026-03-14 09:19:09.638195 | controller |              "^(cifmw|pre|post)_(?!install_yamls|openshift_token|openshift_login|openshift_kubeconfig).*") |
2026-03-14 09:19:09.638199 | controller |   list | items2dict
2026-03-14 09:19:09.638204 | controller | }}] ***
2026-03-14 09:19:09.638208 | controller | Saturday 14 March 2026  09:19:09 +0000 (0:00:00.044)       0:00:01.262 ********
2026-03-14 09:19:09.638213 | controller | Saturday 14 March 2026  09:19:09 +0000 (0:00:00.044)       0:00:01.261 ********
2026-03-14 09:19:09.638223 | controller | ok: [controller]
2026-03-14 09:19:09.955407 | controller |
2026-03-14 09:19:09.955439 | controller | TASK [install_ca : Ensure target directory exists path={{ cifmw_install_ca_trust_dir }}, state=directory, mode=0755] ***
2026-03-14 09:19:09.955445 | controller | Saturday 14 March 2026  09:19:09 +0000 (0:00:00.104)       0:00:01.366 ********
2026-03-14 09:19:09.955451 | controller | Saturday 14 March 2026  09:19:09 +0000 (0:00:00.104)       0:00:01.365 ********
2026-03-14 09:19:09.955460 | controller | ok: [controller]
2026-03-14 09:19:09.987433 | controller |
2026-03-14 09:19:09.987484 | controller | TASK [install_ca : Install internal CA from url url={{ cifmw_install_ca_url }}, dest={{ cifmw_install_ca_trust_dir }}, validate_certs={{ cifmw_install_ca_url_validate_certs | default(omit) }}, mode=0644] ***
2026-03-14 09:19:09.987498 | controller | Saturday 14 March 2026  09:19:09 +0000 (0:00:00.318)       0:00:01.685 ********
2026-03-14 09:19:09.987509 | controller | Saturday 14 March 2026  09:19:09 +0000 (0:00:00.318)       0:00:01.683 ********
2026-03-14 09:19:09.987532 | controller | skipping: [controller]
2026-03-14 09:19:10.014263 | controller |
2026-03-14 09:19:10.014312 | controller | TASK [install_ca : Install custom CA bundle from inline dest={{ cifmw_install_ca_trust_dir }}/cifmw_inline_ca_bundle.crt, content={{ cifmw_install_ca_bundle_inline }}, mode=0644] ***
2026-03-14 09:19:10.014326 | controller | Saturday 14 March 2026  09:19:09 +0000 (0:00:00.031)       0:00:01.716 ********
2026-03-14 09:19:10.014337 | controller | Saturday 14 March 2026  09:19:09 +0000 (0:00:00.031)       0:00:01.715 ********
2026-03-14 09:19:10.014354 | controller | skipping: [controller]
2026-03-14 09:19:10.040156 | controller |
2026-03-14 09:19:10.040198 | controller | TASK [install_ca : Install custom CA bundle from file dest={{ cifmw_install_ca_trust_dir }}/{{ cifmw_install_ca_bundle_src | basename }}, src={{ cifmw_install_ca_bundle_src }}, mode=0644] ***
2026-03-14 09:19:10.040208 | controller | Saturday 14 March 2026  09:19:10 +0000 (0:00:00.026)       0:00:01.743 ********
2026-03-14 09:19:10.040215 | controller | Saturday 14 March 2026  09:19:10 +0000 (0:00:00.026)       0:00:01.742 ********
2026-03-14 09:19:10.040229 | controller | skipping: [controller]
2026-03-14 09:19:10.071595 | controller |
2026-03-14 09:19:10.071667 | controller | TASK [install_ca : Update ca bundle _raw_params=update-ca-trust] ***************
2026-03-14 09:19:10.071680 | controller | Saturday 14 March 2026  09:19:10 +0000 (0:00:00.026)       0:00:01.769 ********
2026-03-14 09:19:10.071697 | controller | Saturday 14 March 2026  09:19:10 +0000 (0:00:00.026)       0:00:01.768 ********
2026-03-14 09:19:10.071711 | controller | skipping: [controller]
2026-03-14 09:19:10.706157 | controller |
2026-03-14 09:19:10.706204 | controller | TASK [repo_setup : Ensure directories are present path={{ cifmw_repo_setup_basedir }}/{{ item }}, state=directory, mode=0755] ***
2026-03-14 09:19:10.706211 | controller | Saturday 14 March 2026  09:19:10 +0000 (0:00:00.031)       0:00:01.801 ********
2026-03-14 09:19:10.706216 | controller | Saturday 14 March 2026  09:19:10 +0000 (0:00:00.031)       0:00:01.799 ********
2026-03-14 09:19:10.706227 | controller | changed: [controller] => (item=tmp)
2026-03-14 09:19:11.981491 | controller | changed: [controller] => (item=artifacts/repositories)
2026-03-14 09:19:11.981542 | controller | changed: [controller] => (item=venv/repo_setup)
2026-03-14 09:19:11.981548 | controller |
2026-03-14 09:19:11.981553 | controller | TASK [repo_setup : Make sure git-core package is installed name=git-core, state=present] ***
2026-03-14 09:19:11.981558 | controller | Saturday 14 March 2026  09:19:10 +0000 (0:00:00.634)       0:00:02.435 ********
2026-03-14 09:19:11.981563 | controller | Saturday 14 March 2026  09:19:10 +0000 (0:00:00.634)       0:00:02.434 ********
2026-03-14 09:19:11.981575 | controller | ok: [controller]
2026-03-14 09:19:12.926246 | controller |
2026-03-14 09:19:12.926288 | controller | TASK [repo_setup : Get repo-setup repository accept_hostkey=True, dest={{ cifmw_repo_setup_basedir }}/tmp/repo-setup, repo={{ cifmw_repo_setup_src }}] ***
2026-03-14 09:19:12.926297 | controller | Saturday 14 March 2026  09:19:11 +0000 (0:00:01.275)       0:00:03.711 ********
2026-03-14 09:19:12.926305 | controller | Saturday 14 March 2026  09:19:11 +0000 (0:00:01.275)       0:00:03.709 ********
2026-03-14 09:19:12.926323 | controller | changed: [controller]
2026-03-14 09:19:20.670197 | controller |
2026-03-14 09:19:20.670302 | controller | TASK [repo_setup : Initialize python venv and install requirements virtualenv={{ cifmw_repo_setup_venv }}, requirements={{ cifmw_repo_setup_basedir }}/tmp/repo-setup/requirements.txt, virtualenv_command=python3 -m venv  --system-site-packages --upgrade-deps] ***
2026-03-14 09:19:20.670324 | controller | Saturday 14 March 2026  09:19:12 +0000 (0:00:00.944)       0:00:04.655 ********
2026-03-14 09:19:20.670331 | controller | Saturday 14 March 2026  09:19:12 +0000 (0:00:00.944)       0:00:04.654 ********
2026-03-14 09:19:20.670364 | controller | changed: [controller]
2026-03-14 09:19:21.559378 | controller |
2026-03-14 09:19:21.559468 | controller | TASK [repo_setup : Install repo-setup package chdir={{ cifmw_repo_setup_basedir }}/tmp/repo-setup, creates={{ cifmw_repo_setup_venv }}/bin/repo-setup, _raw_params={{ cifmw_repo_setup_venv }}/bin/python setup.py install] ***
2026-03-14 09:19:21.559479 | controller | Saturday 14 March 2026  09:19:20 +0000 (0:00:07.743)       0:00:12.399 ********
2026-03-14 09:19:21.559503 | controller | Saturday 14 March 2026  09:19:20 +0000 (0:00:07.743)       0:00:12.397 ********
2026-03-14 09:19:21.559516 | controller | changed: [controller]
2026-03-14 09:19:21.582985 | controller |
2026-03-14 09:19:21.583028 | controller | TASK [repo_setup : Set cifmw_repo_setup_dlrn_hash_tag from content provider cifmw_repo_setup_dlrn_hash_tag={{ content_provider_dlrn_md5_hash }}] ***
2026-03-14 09:19:21.583035 | controller | Saturday 14 March 2026  09:19:21 +0000 (0:00:00.889)       0:00:13.289 ********
2026-03-14 09:19:21.583040 | controller | Saturday 14 March 2026  09:19:21 +0000 (0:00:00.889)       0:00:13.287 ********
2026-03-14 09:19:21.583050 | controller | skipping: [controller]
2026-03-14 09:19:22.126701 | controller |
2026-03-14 09:19:22.126741 | controller | TASK [repo_setup : Run repo-setup _raw_params={{ cifmw_repo_setup_venv }}/bin/repo-setup {{ cifmw_repo_setup_promotion }} {{ cifmw_repo_setup_additional_repos }} -d {{ cifmw_repo_setup_os_release }}{{ cifmw_repo_setup_dist_major_version }} -b {{ cifmw_repo_setup_branch }} --rdo-mirror {{ cifmw_repo_setup_rdo_mirror }} {% if cifmw_repo_setup_dlrn_hash_tag | length > 0 %} --dlrn-hash-tag {{ cifmw_repo_setup_dlrn_hash_tag }} {% endif %} -o {{ cifmw_repo_setup_output }}] ***
2026-03-14 09:19:22.126752 | controller | Saturday 14 March 2026  09:19:21 +0000 (0:00:00.023)       0:00:13.312 ********
2026-03-14 09:19:22.126759 | controller | Saturday 14 March 2026  09:19:21 +0000 (0:00:00.023)       0:00:13.311 ********
2026-03-14 09:19:22.126776 | controller | changed: [controller]
2026-03-14 09:19:22.154666 | controller |
2026-03-14 09:19:22.154701 | controller | TASK [repo_setup : Get component repo url={{ cifmw_repo_setup_dlrn_uri }}/{{ cifmw_repo_setup_os_release }}{{ cifmw_repo_setup_dist_major_version }}-{{ cifmw_repo_setup_branch }}/component/{{ cifmw_repo_setup_component_name }}/{{ cifmw_repo_setup_component_promotion_tag }}/delorean.repo, dest={{ cifmw_repo_setup_output }}/{{ cifmw_repo_setup_component_name }}_{{ cifmw_repo_setup_component_promotion_tag }}_delorean.repo, mode=0644] ***
2026-03-14 09:19:22.154708 | controller | Saturday 14 March 2026  09:19:22 +0000 (0:00:00.543)       0:00:13.856 ********
2026-03-14 09:19:22.154714 | controller | Saturday 14 March 2026  09:19:22 +0000 (0:00:00.543)       0:00:13.854 ********
2026-03-14 09:19:22.154722 | controller | skipping: [controller]
2026-03-14 09:19:22.183540 | controller |
2026-03-14 09:19:22.183579 | controller | TASK [repo_setup : Rename component repo path={{ cifmw_repo_setup_output }}/{{ cifmw_repo_setup_component_name }}_{{ cifmw_repo_setup_component_promotion_tag }}_delorean.repo, regexp=delorean-component-{{ cifmw_repo_setup_component_name }}, replace={{ cifmw_repo_setup_component_name }}-{{ cifmw_repo_setup_component_promotion_tag }}] ***
2026-03-14 09:19:22.183587 | controller | Saturday 14 March 2026  09:19:22 +0000 (0:00:00.027)       0:00:13.884 ********
2026-03-14 09:19:22.183593 | controller | Saturday 14 March 2026  09:19:22 +0000 (0:00:00.027)       0:00:13.882 ********
2026-03-14 09:19:22.183619 | controller | skipping: [controller]
2026-03-14 09:19:22.211101 | controller |
2026-03-14 09:19:22.211136 | controller | TASK [repo_setup : Disable component repo in current-podified dlrn repo path={{ cifmw_repo_setup_output }}/delorean.repo, section=delorean-component-{{ cifmw_repo_setup_component_name }}, option=enabled, value=0, mode=0644] ***
2026-03-14 09:19:22.211143 | controller | Saturday 14 March 2026  09:19:22 +0000 (0:00:00.028)       0:00:13.913 ********
2026-03-14 09:19:22.211148 | controller | Saturday 14 March 2026  09:19:22 +0000 (0:00:00.028)       0:00:13.911 ********
2026-03-14 09:19:22.211157 | controller | skipping: [controller]
2026-03-14 09:19:22.694571 | controller |
2026-03-14 09:19:22.694669 | controller | TASK [repo_setup : Run repo-setup-get-hash _raw_params={{ cifmw_repo_setup_venv }}/bin/repo-setup-get-hash --dlrn-url {{ cifmw_repo_setup_dlrn_uri[:-1] }} --os-version {{ cifmw_repo_setup_os_release }}{{ cifmw_repo_setup_dist_major_version }} --release {{ cifmw_repo_setup_branch }} {% if cifmw_repo_setup_component_name | length > 0 -%} --component {{ cifmw_repo_setup_component_name }} --tag {{ cifmw_repo_setup_component_promotion_tag }} {% else -%} --tag {{cifmw_repo_setup_promotion }} {% endif -%} {% if (cifmw_repo_setup_dlrn_hash_tag | length > 0) and (cifmw_repo_setup_component_name | length <= 0) -%} --dlrn-hash-tag {{ cifmw_repo_setup_dlrn_hash_tag }} {% endif -%} --json] ***
2026-03-14 09:19:22.694686 | controller | Saturday 14 March 2026  09:19:22 +0000 (0:00:00.027)       0:00:13.941 ********
2026-03-14 09:19:22.694702 | controller | Saturday 14 March 2026  09:19:22 +0000 (0:00:00.027)       0:00:13.939 ********
2026-03-14 09:19:22.694720 | controller | changed: [controller]
2026-03-14 09:19:23.587070 | controller |
2026-03-14 09:19:23.587141 | controller | TASK [repo_setup : Dump full hash in delorean.repo.md5 file content={{ _repo_setup_json['full_hash'] }}
2026-03-14 09:19:23.587154 | controller | , dest={{ cifmw_repo_setup_basedir }}/artifacts/repositories/delorean.repo.md5, mode=0644] ***
2026-03-14 09:19:23.587173 | controller | Saturday 14 March 2026  09:19:22 +0000 (0:00:00.482)       0:00:14.423 ********
2026-03-14 09:19:23.587184 | controller | Saturday 14 March 2026  09:19:22 +0000 (0:00:00.482)       0:00:14.421 ********
2026-03-14 09:19:23.587203 | controller | changed: [controller]
2026-03-14 09:19:23.613019 | controller |
2026-03-14 09:19:23.613096 | controller | TASK [repo_setup : Dump current-podified hash url={{ cifmw_repo_setup_dlrn_uri }}/{{ cifmw_repo_setup_os_release }}{{ cifmw_repo_setup_dist_major_version }}-{{ cifmw_repo_setup_branch }}/current-podified/delorean.repo.md5, dest={{ cifmw_repo_setup_basedir }}/artifacts/repositories/delorean.repo.md5, mode=0644] ***
2026-03-14 09:19:23.613112 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.891)       0:00:15.315 ********
2026-03-14 09:19:23.613137 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.891)       0:00:15.313 ********
2026-03-14 09:19:23.613194 | controller | skipping: [controller]
2026-03-14 09:19:23.639367 | controller |
2026-03-14 09:19:23.639434 | controller | TASK [repo_setup : Slurp current podified hash src={{ cifmw_repo_setup_basedir }}/artifacts/repositories/delorean.repo.md5] ***
2026-03-14 09:19:23.639445 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.027)       0:00:15.342 ********
2026-03-14 09:19:23.639454 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.027)       0:00:15.341 ********
2026-03-14 09:19:23.639471 | controller | skipping: [controller]
2026-03-14 09:19:23.661145 | controller |
2026-03-14 09:19:23.661220 | controller | TASK [repo_setup : Update the value of full_hash _repo_setup_json={{ _repo_setup_json | combine({'full_hash': _hash}, recursive=true) }}] ***
2026-03-14 09:19:23.661230 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.026)       0:00:15.369 ********
2026-03-14 09:19:23.661237 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.026)       0:00:15.367 ********
2026-03-14 09:19:23.661252 | controller | skipping: [controller]
2026-03-14 09:19:23.695347 | controller |
2026-03-14 09:19:23.695408 | controller | TASK [repo_setup : Export hashes facts for further use cifmw_repo_setup_full_hash={{ _repo_setup_json['full_hash'] }}, cifmw_repo_setup_commit_hash={{ _repo_setup_json['commit_hash'] }}, cifmw_repo_setup_distro_hash={{ _repo_setup_json['distro_hash'] }}, cifmw_repo_setup_extended_hash={{ _repo_setup_json['extended_hash'] }}, cifmw_repo_setup_dlrn_api_url={{ _repo_setup_json['dlrn_api_url'] }}, cifmw_repo_setup_dlrn_url={{ _repo_setup_json['dlrn_url'] }}, cifmw_repo_setup_release={{ _repo_setup_json['release'] }}, cacheable=True] ***
2026-03-14 09:19:23.695419 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.021)       0:00:15.390 ********
2026-03-14 09:19:23.695426 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.021)       0:00:15.389 ********
2026-03-14 09:19:23.695440 | controller | ok: [controller]
2026-03-14 09:19:23.714431 | controller |
2026-03-14 09:19:23.714491 | controller | TASK [repo_setup : Create download directory path={{ cifmw_repo_setup_rhos_release_path }}, state=directory, mode=0755] ***
2026-03-14 09:19:23.714500 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.034)       0:00:15.425 ********
2026-03-14 09:19:23.714507 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.034)       0:00:15.423 ********
2026-03-14 09:19:23.714522 | controller | skipping: [controller]
2026-03-14 09:19:23.732657 | controller |
2026-03-14 09:19:23.732712 | controller | TASK [repo_setup : Print the URL to request msg={{ cifmw_repo_setup_rhos_release_rpm }}] ***
2026-03-14 09:19:23.732724 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.018)       0:00:15.444 ********
2026-03-14 09:19:23.732732 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.019)       0:00:15.442 ********
2026-03-14 09:19:23.732749 | controller | skipping: [controller]
2026-03-14 09:19:23.755060 | controller |
2026-03-14 09:19:23.755098 | controller | TASK [Download the RPM name=krb_request] ***************************************
2026-03-14 09:19:23.755107 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.018)       0:00:15.462 ********
2026-03-14 09:19:23.755113 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.018)       0:00:15.460 ********
2026-03-14 09:19:23.755125 | controller | skipping: [controller]
2026-03-14 09:19:23.773701 | controller |
2026-03-14 09:19:23.773733 | controller | TASK [repo_setup : Install RHOS Release tool name={{
2026-03-14 09:19:23.773743 | controller |   cifmw_repo_setup_rhos_release_rpm
2026-03-14 09:19:23.773749 | controller |   if cifmw_repo_setup_rhos_release_rpm is not url
2026-03-14 09:19:23.773755 | controller |   else cifmw_krb_request_out.path
2026-03-14 09:19:23.773762 | controller | }}, state=present, disable_gpg_check={{ cifmw_repo_setup_rhos_release_gpg_check | bool }}] ***
2026-03-14 09:19:23.773769 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.022)       0:00:15.484 ********
2026-03-14 09:19:23.773775 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.022)       0:00:15.483 ********
2026-03-14 09:19:23.773786 | controller | skipping: [controller]
2026-03-14 09:19:23.793407 | controller |
2026-03-14 09:19:23.793441 | controller | TASK [repo_setup : Get rhos-release tool version _raw_params=rhos-release --version] ***
2026-03-14 09:19:23.793457 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.018)       0:00:15.503 ********
2026-03-14 09:19:23.793464 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.018)       0:00:15.501 ********
2026-03-14 09:19:23.793475 | controller | skipping: [controller]
2026-03-14 09:19:23.815330 | controller |
2026-03-14 09:19:23.815370 | controller | TASK [repo_setup : Print rhos-release tool version msg={{ rr_version.stdout }}] ***
2026-03-14 09:19:23.815384 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.019)       0:00:15.523 ********
2026-03-14 09:19:23.815391 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.019)       0:00:15.521 ********
2026-03-14 09:19:23.815403 | controller | skipping: [controller]
2026-03-14 09:19:23.835159 | controller |
2026-03-14 09:19:23.835200 | controller | TASK [repo_setup : Generate repos using rhos-release {{ cifmw_repo_setup_rhos_release_args }} _raw_params=rhos-release {{ cifmw_repo_setup_rhos_release_args }} \
2026-03-14 09:19:23.835211 | controller |   -t {{ cifmw_repo_setup_output }}] ***
2026-03-14 09:19:23.835221 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.021)       0:00:15.544 ********
2026-03-14 09:19:23.835229 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.021)       0:00:15.543 ********
2026-03-14 09:19:23.835244 | controller | skipping: [controller]
2026-03-14 09:19:24.071545 | controller |
2026-03-14 09:19:24.071639 | controller | TASK [repo_setup : Check for /etc/ci/mirror_info.sh path=/etc/ci/mirror_info.sh] ***
2026-03-14 09:19:24.071659 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.019)       0:00:15.564 ********
2026-03-14 09:19:24.071671 | controller | Saturday 14 March 2026  09:19:23 +0000 (0:00:00.019)       0:00:15.563 ********
2026-03-14 09:19:24.071691 | controller | ok: [controller]
2026-03-14 09:19:24.292111 | controller |
2026-03-14 09:19:24.292165 | controller | TASK [repo_setup : Use RDO proxy mirrors chdir={{ cifmw_repo_setup_output }}, _raw_params=set -o pipefail
2026-03-14 09:19:24.292178 | controller | source /etc/ci/mirror_info.sh
2026-03-14 09:19:24.292189 | controller | sed -i -e "s|https://trunk.rdoproject.org|$NODEPOOL_RDO_PROXY|g" *.repo
2026-03-14 09:19:24.292200 | controller | ] ***
2026-03-14 09:19:24.292210 | controller | Saturday 14 March 2026  09:19:24 +0000 (0:00:00.235)       0:00:15.800 ********
2026-03-14 09:19:24.292220 | controller | Saturday 14 March 2026  09:19:24 +0000 (0:00:00.235)       0:00:15.799 ********
2026-03-14 09:19:24.292239 | controller | changed: [controller]
2026-03-14 09:19:24.558499 | controller |
2026-03-14 09:19:24.558536 | controller | TASK [repo_setup : Use RDO CentOS mirrors (remove CentOS 10 conditional when Nodepool mirrors exist) chdir={{ cifmw_repo_setup_output }}, _raw_params=set -o pipefail
2026-03-14 09:19:24.558543 | controller | source /etc/ci/mirror_info.sh
2026-03-14 09:19:24.558548 | controller | sed -i -e "s|http://mirror.stream.centos.org|$NODEPOOL_CENTOS_MIRROR|g" *.repo
2026-03-14 09:19:24.558553 | controller | ] ***
2026-03-14 09:19:24.558562 | controller | Saturday 14 March 2026  09:19:24 +0000 (0:00:00.221)       0:00:16.022 ********
2026-03-14 09:19:24.558566 | controller | Saturday 14 March 2026  09:19:24 +0000 (0:00:00.221)       0:00:16.020 ********
2026-03-14 09:19:24.558575 | controller | changed: [controller]
2026-03-14 09:19:25.133841 | controller |
2026-03-14 09:19:25.133887 | controller | TASK [repo_setup : Check for gating.repo file on content provider url=http://{{ content_provider_registry_ip }}:8766/gating.repo] ***
2026-03-14 09:19:25.133894 | controller | Saturday 14 March 2026  09:19:24 +0000 (0:00:00.266)       0:00:16.288 ********
2026-03-14 09:19:25.133899 | controller | Saturday 14 March 2026  09:19:24 +0000 (0:00:00.266)       0:00:16.286 ********
2026-03-14 09:19:25.133913 | controller | task path: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/repo_setup/tasks/populate_gating_repo.yml:2
2026-03-14 09:19:25.175737 | controller | fatal: [controller]: FAILED! =>
2026-03-14 09:19:25.175770 | controller |     changed: false
2026-03-14 09:19:25.175776 | controller |     elapsed: 0
2026-03-14 09:19:25.175782 | controller |     msg: 'Status code was -1 and not [200]: Request failed: <urlopen error [Errno 111]
2026-03-14 09:19:25.175787 | controller |       Connection refused>'
2026-03-14 09:19:25.175792 | controller |     redirected: false
2026-03-14 09:19:25.175796 | controller |     status: -1
2026-03-14 09:19:25.175804 | controller |     url: http://38.102.83.243:8766/gating.repo
2026-03-14 09:19:25.175815 | controller | ...ignoring
2026-03-14 09:19:25.175820 | controller |
2026-03-14 09:19:25.175824 | controller | TASK [repo_setup : Populate gating repo from content provider ip content=[gating-repo]
2026-03-14 09:19:25.175829 | controller | baseurl=http://{{ content_provider_registry_ip }}:8766/
2026-03-14 09:19:25.175833 | controller | enabled=1
2026-03-14 09:19:25.175839 | controller | gpgcheck=0
2026-03-14 09:19:25.175844 | controller | priority=1
2026-03-14 09:19:25.175848 | controller | , dest={{ cifmw_repo_setup_output }}/gating.repo, mode=0644] ***
2026-03-14 09:19:25.175852 | controller | Saturday 14 March 2026  09:19:25 +0000 (0:00:00.575)       0:00:16.863 ********
2026-03-14 09:19:25.175857 | controller | Saturday 14 March 2026  09:19:25 +0000 (0:00:00.575)       0:00:16.862 ********
2026-03-14 09:19:25.175866 | controller | skipping: [controller]
2026-03-14 09:19:25.209516 | controller |
2026-03-14 09:19:25.209552 | controller | TASK [repo_setup : Check for DLRN repo at the destination path={{ cifmw_repo_setup_output }}/delorean.repo] ***
2026-03-14 09:19:25.209558 | controller | Saturday 14 March 2026  09:19:25 +0000 (0:00:00.041)       0:00:16.905 ********
2026-03-14 09:19:25.209563 | controller | Saturday 14 March 2026  09:19:25 +0000 (0:00:00.041)       0:00:16.903 ********
2026-03-14 09:19:25.209572 | controller | skipping: [controller]
2026-03-14 09:19:25.241258 | controller |
2026-03-14 09:19:25.241292 | controller | TASK [repo_setup : Lower the priority of DLRN repos to allow installation from gating repo path={{ cifmw_repo_setup_output }}/delorean.repo, regexp=priority=1, replace=priority=20] ***
2026-03-14 09:19:25.241298 | controller | Saturday 14 March 2026  09:19:25 +0000 (0:00:00.033)       0:00:16.939 ********
2026-03-14 09:19:25.241303 | controller | Saturday 14 March 2026  09:19:25 +0000 (0:00:00.033)       0:00:16.937 ********
2026-03-14 09:19:25.241313 | controller | skipping: [controller]
2026-03-14 09:19:25.273963 | controller |
2026-03-14 09:19:25.273998 | controller | TASK [repo_setup : Check for DLRN component repo path={{ cifmw_repo_setup_output }}/{{ _comp_repo }}] ***
2026-03-14 09:19:25.274005 | controller | Saturday 14 March 2026  09:19:25 +0000 (0:00:00.031)       0:00:16.971 ********
2026-03-14 09:19:25.274010 | controller | Saturday 14 March 2026  09:19:25 +0000 (0:00:00.031)       0:00:16.969 ********
2026-03-14 09:19:25.274019 | controller | skipping: [controller]
2026-03-14 09:19:25.307088 | controller |
2026-03-14 09:19:25.307125 | controller | TASK [repo_setup : Lower the priority of component repos to allow installation from gating repo path={{ cifmw_repo_setup_output }}/{{ _comp_repo }}, regexp=priority=1, replace=priority=2] ***
2026-03-14 09:19:25.307132 | controller | Saturday 14 March 2026  09:19:25 +0000 (0:00:00.032)       0:00:17.003 ********
2026-03-14 09:19:25.307138 | controller | Saturday 14 March 2026  09:19:25 +0000 (0:00:00.032)       0:00:17.002 ********
2026-03-14 09:19:25.307148 | controller | skipping: [controller]
2026-03-14 09:19:25.650938 | controller |
2026-03-14 09:19:25.650973 | controller | TASK [repo_setup : Find existing repos from /etc/yum.repos.d directory paths=/etc/yum.repos.d/, patterns=*.repo, recurse=False] ***
2026-03-14 09:19:25.650984 | controller | Saturday 14 March 2026  09:19:25 +0000 (0:00:00.033)       0:00:17.036 ********
2026-03-14 09:19:25.650989 | controller | Saturday 14 March 2026  09:19:25 +0000 (0:00:00.033)       0:00:17.035 ********
2026-03-14 09:19:25.651003 | controller | ok: [controller]
2026-03-14 09:19:26.041826 | controller |
2026-03-14 09:19:26.041876 | controller | TASK [repo_setup : Remove existing repos from /etc/yum.repos.d directory path={{ item }}, state=absent] ***
2026-03-14 09:19:26.041890 | controller | Saturday 14 March 2026  09:19:25 +0000 (0:00:00.343)       0:00:17.380 ********
2026-03-14 09:19:26.041900 | controller | Saturday 14 March 2026  09:19:25 +0000 (0:00:00.343)       0:00:17.379 ********
2026-03-14 09:19:26.041918 | controller | changed: [controller] => (item=/etc/yum.repos.d/centos-addons.repo)
2026-03-14 09:19:26.442435 | controller | changed: [controller] => (item=/etc/yum.repos.d/centos.repo)
2026-03-14 09:19:26.442471 | controller |
2026-03-14 09:19:26.442477 | controller | TASK [repo_setup : Cleanup existing metadata _raw_params=dnf clean metadata] ***
2026-03-14 09:19:26.442482 | controller | Saturday 14 March 2026  09:19:26 +0000 (0:00:00.390)       0:00:17.771 ********
2026-03-14 09:19:26.442492 | controller | Saturday 14 March 2026  09:19:26 +0000 (0:00:00.390)       0:00:17.769 ********
2026-03-14 09:19:26.442502 | controller | changed: [controller]
2026-03-14 09:19:26.745203 | controller |
2026-03-14 09:19:26.745237 | controller | TASK [repo_setup : Copy generated repos to /etc/yum.repos.d directory mode=0755, remote_src=True, src={{ cifmw_repo_setup_output }}/, dest=/etc/yum.repos.d] ***
2026-03-14 09:19:26.745244 | controller | Saturday 14 March 2026  09:19:26 +0000 (0:00:00.400)       0:00:18.171 ********
2026-03-14 09:19:26.745249 | controller | Saturday 14 March 2026  09:19:26 +0000 (0:00:00.400)       0:00:18.170 ********
2026-03-14 09:19:26.745258 | controller | changed: [controller]
2026-03-14 09:19:26.796591 | controller |
2026-03-14 09:19:26.796643 | controller | TASK [ci_setup : Gather variables for each operating system _raw_params={{ item }}] ***
2026-03-14 09:19:26.796650 | controller | Saturday 14 March 2026  09:19:26 +0000 (0:00:00.303)       0:00:18.474 ********
2026-03-14 09:19:26.796655 | controller | Saturday 14 March 2026  09:19:26 +0000 (0:00:00.303)       0:00:18.473 ********
2026-03-14 09:19:26.796664 | controller | ok: [controller] => (item=/home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/ci_setup/vars/redhat.yml)
2026-03-14 09:19:26.818726 | controller |
2026-03-14 09:19:26.818759 | controller | TASK [ci_setup : List packages to install var=cifmw_ci_setup_packages] *********
2026-03-14 09:19:26.818765 | controller | Saturday 14 March 2026  09:19:26 +0000 (0:00:00.051)       0:00:18.526 ********
2026-03-14 09:19:26.818770 | controller | Saturday 14 March 2026  09:19:26 +0000 (0:00:00.051)       0:00:18.524 ********
2026-03-14 09:19:26.818779 | controller | ok: [controller] =>
2026-03-14 09:19:54.595881 | controller |     cifmw_ci_setup_packages:
2026-03-14 09:19:54.595917 | controller |     - bash-completion
2026-03-14 09:19:54.595924 | controller |     - ca-certificates
2026-03-14 09:19:54.595929 | controller |     - git-core
2026-03-14 09:19:54.595933 | controller |     - make
2026-03-14 09:19:54.595938 | controller |     - tar
2026-03-14 09:19:54.595942 | controller |     - tmux
2026-03-14 09:19:54.595946 | controller |     - python3-pip
2026-03-14 09:19:54.595951 | controller |
2026-03-14 09:19:54.595956 | controller | TASK [ci_setup : Install needed packages name={{ cifmw_ci_setup_packages }}, state=latest] ***
2026-03-14 09:19:54.595961 | controller | Saturday 14 March 2026  09:19:26 +0000 (0:00:00.022)       0:00:18.548 ********
2026-03-14 09:19:54.595966 | controller | Saturday 14 March 2026  09:19:26 +0000 (0:00:00.022)       0:00:18.547 ********
2026-03-14 09:19:54.595976 | controller | changed: [controller]
2026-03-14 09:19:54.799987 | controller |
2026-03-14 09:19:54.800027 | controller | TASK [ci_setup : Gather version of openshift client _raw_params=oc version --client -o yaml] ***
2026-03-14 09:19:54.800036 | controller | Saturday 14 March 2026  09:19:54 +0000 (0:00:27.777)       0:00:46.325 ********
2026-03-14 09:19:54.800043 | controller | Saturday 14 March 2026  09:19:54 +0000 (0:00:27.777)       0:00:46.324 ********
2026-03-14 09:19:54.800052 | controller | ok: [controller]
2026-03-14 09:19:55.007034 | controller |
2026-03-14 09:19:55.007067 | controller | TASK [ci_setup : Ensure openshift client install path is present path={{ cifmw_ci_setup_oc_install_path }}, state=directory, mode=0755] ***
2026-03-14 09:19:55.007074 | controller | Saturday 14 March 2026  09:19:54 +0000 (0:00:00.204)       0:00:46.529 ********
2026-03-14 09:19:55.007079 | controller | Saturday 14 March 2026  09:19:54 +0000 (0:00:00.204)       0:00:46.528 ********
2026-03-14 09:19:55.007087 | controller | changed: [controller]
2026-03-14 09:20:00.472963 | controller |
2026-03-14 09:20:00.473042 | controller | TASK [ci_setup : Install openshift client src={{ cifmw_ci_setup_openshift_client_download_uri }}/{{ cifmw_ci_setup_openshift_client_version }}/openshift-client-linux.tar.gz, dest={{ cifmw_ci_setup_oc_install_path }}, remote_src=True, mode=0755, creates={{ cifmw_ci_setup_oc_install_path }}/oc] ***
2026-03-14 09:20:00.473055 | controller | Saturday 14 March 2026  09:19:55 +0000 (0:00:00.207)       0:00:46.736 ********
2026-03-14 09:20:00.473062 | controller | Saturday 14 March 2026  09:19:55 +0000 (0:00:00.207)       0:00:46.735 ********
2026-03-14 09:20:00.473073 | controller | changed: [controller]
2026-03-14 09:20:00.498173 | controller |
2026-03-14 09:20:00.498215 | controller | TASK [ci_setup : Add the OC path to cifmw_path if needed cifmw_path={{ cifmw_ci_setup_oc_install_path }}:{{ ansible_env.PATH }}, cacheable=True] ***
2026-03-14 09:20:00.498225 | controller | Saturday 14 March 2026  09:20:00 +0000 (0:00:05.465)       0:00:52.202 ********
2026-03-14 09:20:00.498233 | controller | Saturday 14 March 2026  09:20:00 +0000 (0:00:05.465)       0:00:52.201 ********
2026-03-14 09:20:00.498252 | controller | skipping: [controller]
2026-03-14 09:20:00.783090 | controller |
2026-03-14 09:20:00.783147 | controller | TASK [ci_setup : Create completion file] ***************************************
2026-03-14 09:20:00.783156 | controller | Saturday 14 March 2026  09:20:00 +0000 (0:00:00.025)       0:00:52.227 ********
2026-03-14 09:20:00.783161 | controller | Saturday 14 March 2026  09:20:00 +0000 (0:00:00.025)       0:00:52.226 ********
2026-03-14 09:20:00.783177 | controller | changed: [controller]
2026-03-14 09:20:01.091876 | controller |
2026-03-14 09:20:01.091915 | controller | TASK [ci_setup : Source completion from within .bashrc create=True, mode=0644, path={{ ansible_user_dir }}/.bashrc, block=if [ -f ~/.oc_completion ]; then
2026-03-14 09:20:01.091923 | controller |   source ~/.oc_completion
2026-03-14 09:20:01.091930 | controller | fi] ***
2026-03-14 09:20:01.091936 | controller | Saturday 14 March 2026  09:20:00 +0000 (0:00:00.284)       0:00:52.512 ********
2026-03-14 09:20:01.091942 | controller | Saturday 14 March 2026  09:20:00 +0000 (0:00:00.284)       0:00:52.511 ********
2026-03-14 09:20:01.091953 | controller | changed: [controller]
2026-03-14 09:20:01.108258 | controller |
2026-03-14 09:20:01.108294 | controller | TASK [ci_setup : Check rhsm status _raw_params=subscription-manager status] ****
2026-03-14 09:20:01.108301 | controller | Saturday 14 March 2026  09:20:01 +0000 (0:00:00.308)       0:00:52.821 ********
2026-03-14 09:20:01.108306 | controller | Saturday 14 March 2026  09:20:01 +0000 (0:00:00.308)       0:00:52.820 ********
2026-03-14 09:20:01.108315 | controller | skipping: [controller]
2026-03-14 09:20:01.123266 | controller |
2026-03-14 09:20:01.123301 | controller | TASK [ci_setup : Gather the repos to be enabled _repos={{
2026-03-14 09:20:01.123308 | controller |   cifmw_ci_setup_rhel_rhsm_default_repos +
2026-03-14 09:20:01.123313 | controller |   (cifmw_ci_setup_rhel_rhsm_extra_repos | default([]))
2026-03-14 09:20:01.123317 | controller | }}] ***
2026-03-14 09:20:01.123322 | controller | Saturday 14 March 2026  09:20:01 +0000 (0:00:00.016)       0:00:52.838 ********
2026-03-14 09:20:01.123327 | controller | Saturday 14 March 2026  09:20:01 +0000 (0:00:00.016)       0:00:52.836 ********
2026-03-14 09:20:01.123335 | controller | skipping: [controller]
2026-03-14 09:20:01.139657 | controller |
2026-03-14 09:20:01.139690 | controller | TASK [ci_setup : Enabling the required repositories. name={{ item }}, state={{ rhsm_repo_state | default('enabled') }}] ***
2026-03-14 09:20:01.139696 | controller | Saturday 14 March 2026  09:20:01 +0000 (0:00:00.014)       0:00:52.853 ********
2026-03-14 09:20:01.139706 | controller | Saturday 14 March 2026  09:20:01 +0000 (0:00:00.014)       0:00:52.851 ********
2026-03-14 09:20:01.139715 | controller | skipping: [controller]
2026-03-14 09:20:01.154807 | controller |
2026-03-14 09:20:01.154839 | controller | TASK [ci_setup : Get current /etc/redhat-release _raw_params=cat /etc/redhat-release] ***
2026-03-14 09:20:01.154846 | controller | Saturday 14 March 2026  09:20:01 +0000 (0:00:00.016)       0:00:52.869 ********
2026-03-14 09:20:01.154851 | controller | Saturday 14 March 2026  09:20:01 +0000 (0:00:00.016)       0:00:52.867 ********
2026-03-14 09:20:01.154860 | controller | skipping: [controller]
2026-03-14 09:20:01.169462 | controller |
2026-03-14 09:20:01.169492 | controller | TASK [ci_setup : Print current /etc/redhat-release msg={{ _current_rh_release.stdout }}] ***
2026-03-14 09:20:01.169499 | controller | Saturday 14 March 2026  09:20:01 +0000 (0:00:00.015)       0:00:52.884 ********
2026-03-14 09:20:01.169505 | controller | Saturday 14 March 2026  09:20:01 +0000 (0:00:00.015)       0:00:52.883 ********
2026-03-14 09:20:01.169514 | controller | skipping: [controller]
2026-03-14 09:20:01.191011 | controller |
2026-03-14 09:20:01.191045 | controller | TASK [ci_setup : Ensure the repos are enabled in the system using yum name={{ item.name }}, baseurl={{ item.baseurl }}, description={{ item.description | default(item.name) }}, gpgcheck={{ item.gpgcheck | default(false) }}, enabled=True, state={{ yum_repo_state | default('present') }}] ***
2026-03-14 09:20:01.191052 | controller | Saturday 14 March 2026  09:20:01 +0000 (0:00:00.014)       0:00:52.899 ********
2026-03-14 09:20:01.191057 | controller | Saturday 14 March 2026  09:20:01 +0000 (0:00:00.014)       0:00:52.897 ********
2026-03-14 09:20:01.191066 | controller | skipping: [controller]
2026-03-14 09:20:02.273917 | controller |
2026-03-14 09:20:02.273973 | controller | TASK [ci_setup : Manage directories path={{ item }}, state={{ directory_state }}, mode=0755, owner={{ ansible_user_id }}, group={{ ansible_user_id }}] ***
2026-03-14 09:20:02.273999 | controller | Saturday 14 March 2026  09:20:01 +0000 (0:00:00.021)       0:00:52.920 ********
2026-03-14 09:20:02.274010 | controller | Saturday 14 March 2026  09:20:01 +0000 (0:00:00.021)       0:00:52.919 ********
2026-03-14 09:20:02.274028 | controller | changed: [controller] => (item=/home/zuul/ci-framework-data/artifacts/manifests/openstack/cr)
2026-03-14 09:20:02.393640 | controller | changed: [controller] => (item=/home/zuul/ci-framework-data/logs)
2026-03-14 09:20:02.393682 | controller | ok: [controller] => (item=/home/zuul/ci-framework-data/tmp)
2026-03-14 09:20:02.393696 | controller | changed: [controller] => (item=/home/zuul/ci-framework-data/volumes)
2026-03-14 09:20:02.393702 | controller | ok: [controller] => (item=/home/zuul/ci-framework-data/artifacts/parameters)
2026-03-14 09:20:02.393708 | controller |
2026-03-14 09:20:02.393714 | controller | TASK [Prepare install_yamls make targets name=install_yamls, apply={'tags': ['bootstrap']}] ***
2026-03-14 09:20:02.393720 | controller | Saturday 14 March 2026  09:20:02 +0000 (0:00:01.082)       0:00:54.003 ********
2026-03-14 09:20:02.393725 | controller | Saturday 14 March 2026  09:20:02 +0000 (0:00:01.082)       0:00:54.002 ********
2026-03-14 09:20:02.393736 | controller |
2026-03-14 09:20:02.997447 | controller | TASK [install_yamls : Ensure directories exist path={{ item }}, state=directory, mode=0755] ***
2026-03-14 09:20:02.997496 | controller | Saturday 14 March 2026  09:20:02 +0000 (0:00:00.119)       0:00:54.123 ********
2026-03-14 09:20:02.997502 | controller | Saturday 14 March 2026  09:20:02 +0000 (0:00:00.119)       0:00:54.121 ********
2026-03-14 09:20:02.997514 | controller | ok: [controller] => (item=/home/zuul/ci-framework-data/artifacts)
2026-03-14 09:20:03.036575 | controller | changed: [controller] => (item=/home/zuul/ci-framework-data/artifacts/roles/install_yamls_makes/tasks)
2026-03-14 09:20:03.036641 | controller | ok: [controller] => (item=/home/zuul/ci-framework-data/artifacts/parameters)
2026-03-14 09:20:03.036647 | controller |
2026-03-14 09:20:03.036652 | controller | TASK [Create variables with local repos based on Zuul items name=install_yamls, tasks_from=zuul_set_operators_repo.yml] ***
2026-03-14 09:20:03.036657 | controller | Saturday 14 March 2026  09:20:02 +0000 (0:00:00.603)       0:00:54.727 ********
2026-03-14 09:20:03.036662 | controller | Saturday 14 March 2026  09:20:02 +0000 (0:00:00.603)       0:00:54.725 ********
2026-03-14 09:20:03.036673 | controller |
2026-03-14 09:20:03.086099 | controller | TASK [install_yamls : Set fact with local repos based on Zuul items cifmw_install_yamls_operators_repo={{ cifmw_install_yamls_operators_repo | default({}) | combine(_repo_operator_info | items2dict) }}] ***
2026-03-14 09:20:03.086150 | controller | Saturday 14 March 2026  09:20:03 +0000 (0:00:00.039)       0:00:54.766 ********
2026-03-14 09:20:03.086156 | controller | Saturday 14 March 2026  09:20:03 +0000 (0:00:00.039)       0:00:54.764 ********
2026-03-14 09:20:03.086170 | controller | ok: [controller] => (item={'branch': 'main', 'change': '352', 'change_url': 'https://github.com/openstack-k8s-operators/watcher-operator/pull/352', 'commit_id': '0d9467a022a6d6a8b6465cdbc9e4aefc533e796a', 'patchset': '0d9467a022a6d6a8b6465cdbc9e4aefc533e796a', 'project': {'canonical_hostname': 'github.com', 'canonical_name': 'github.com/openstack-k8s-operators/watcher-operator', 'name': 'openstack-k8s-operators/watcher-operator', 'short_name': 'watcher-operator', 'src_dir': 'src/github.com/openstack-k8s-operators/watcher-operator'}, 'topic': None})
2026-03-14 09:20:03.144503 | controller |
2026-03-14 09:20:03.144573 | controller | TASK [install_yamls : Print helpful data for debugging msg=_repo_operator_name: {{ _repo_operator_name }}
2026-03-14 09:20:03.144579 | controller | _repo_operator_info: {{ _repo_operator_info }}
2026-03-14 09:20:03.144585 | controller | cifmw_install_yamls_operators_repo: {{ cifmw_install_yamls_operators_repo }}
2026-03-14 09:20:03.144589 | controller | ] ***
2026-03-14 09:20:03.144594 | controller | Saturday 14 March 2026  09:20:03 +0000 (0:00:00.049)       0:00:54.815 ********
2026-03-14 09:20:03.144611 | controller | Saturday 14 March 2026  09:20:03 +0000 (0:00:00.049)       0:00:54.814 ********
2026-03-14 09:20:03.144629 | controller | ok: [controller] => (item={'branch': 'main', 'change': '352', 'change_url': 'https://github.com/openstack-k8s-operators/watcher-operator/pull/352', 'commit_id': '0d9467a022a6d6a8b6465cdbc9e4aefc533e796a', 'patchset': '0d9467a022a6d6a8b6465cdbc9e4aefc533e796a', 'project': {'canonical_hostname': 'github.com', 'canonical_name': 'github.com/openstack-k8s-operators/watcher-operator', 'name': 'openstack-k8s-operators/watcher-operator', 'short_name': 'watcher-operator', 'src_dir': 'src/github.com/openstack-k8s-operators/watcher-operator'}, 'topic': None}) =>
2026-03-14 09:20:03.211380 | controller |     msg: |
2026-03-14 09:20:03.211414 | controller |       _repo_operator_name: watcher
2026-03-14 09:20:03.211421 | controller |       _repo_operator_info: [{'key': 'WATCHER_REPO', 'value': '/home/zuul/src/github.com/openstack-k8s-operators/watcher-operator'}, {'key': 'WATCHER_BRANCH', 'value': ''}]
2026-03-14 09:20:03.211427 | controller |       cifmw_install_yamls_operators_repo: {'WATCHER_REPO': '/home/zuul/src/github.com/openstack-k8s-operators/watcher-operator', 'WATCHER_BRANCH': ''}
2026-03-14 09:20:03.211432 | controller |
2026-03-14 09:20:03.211437 | controller | TASK [Customize install_yamls devsetup vars if needed name=install_yamls, tasks_from=customize_devsetup_vars.yml] ***
2026-03-14 09:20:03.211442 | controller | Saturday 14 March 2026  09:20:03 +0000 (0:00:00.058)       0:00:54.874 ********
2026-03-14 09:20:03.211447 | controller | Saturday 14 March 2026  09:20:03 +0000 (0:00:00.058)       0:00:54.872 ********
2026-03-14 09:20:03.211456 | controller |
2026-03-14 09:20:03.243545 | controller | TASK [install_yamls : Update opm_version in install_yamls devsetup/vars/default.yaml path={{ cifmw_install_yamls_repo }}/devsetup/vars/default.yaml, regexp=^opm_version:, line=opm_version: {{ cifmw_install_yamls_opm_version }}, state=present] ***
2026-03-14 09:20:03.243577 | controller | Saturday 14 March 2026  09:20:03 +0000 (0:00:00.067)       0:00:54.941 ********
2026-03-14 09:20:03.243584 | controller | Saturday 14 March 2026  09:20:03 +0000 (0:00:00.067)       0:00:54.939 ********
2026-03-14 09:20:03.243592 | controller | skipping: [controller]
2026-03-14 09:20:03.275793 | controller |
2026-03-14 09:20:03.275828 | controller | TASK [install_yamls : Update sdk_version in install_yamls devsetup/vars/default.yaml path={{ cifmw_install_yamls_repo }}/devsetup/vars/default.yaml, regexp=^sdk_version:, line=sdk_version: {{ cifmw_install_yamls_sdk_version }}, state=present] ***
2026-03-14 09:20:03.275836 | controller | Saturday 14 March 2026  09:20:03 +0000 (0:00:00.032)       0:00:54.973 ********
2026-03-14 09:20:03.275841 | controller | Saturday 14 March 2026  09:20:03 +0000 (0:00:00.032)       0:00:54.971 ********
2026-03-14 09:20:03.275849 | controller | skipping: [controller]
2026-03-14 09:20:03.308480 | controller |
2026-03-14 09:20:03.308522 | controller | TASK [install_yamls : Update go_version in install_yamls devsetup/vars/default.yaml path={{ cifmw_install_yamls_repo }}/devsetup/vars/default.yaml, regexp=^go_version:, line=go_version: {{ cifmw_install_yamls_go_version }}, state=present] ***
2026-03-14 09:20:03.308533 | controller | Saturday 14 March 2026  09:20:03 +0000 (0:00:00.032)       0:00:55.005 ********
2026-03-14 09:20:03.308541 | controller | Saturday 14 March 2026  09:20:03 +0000 (0:00:00.032)       0:00:55.003 ********
2026-03-14 09:20:03.308554 | controller | skipping: [controller]
2026-03-14 09:20:03.344510 | controller |
2026-03-14 09:20:03.344561 | controller | TASK [install_yamls : Update kustomize_version in install_yamls devsetup/vars/default.yaml path={{ cifmw_install_yamls_repo }}/devsetup/vars/default.yaml, regexp=^kustomize_version:, line=kustomize_version: {{ cifmw_install_yamls_kustomize_version }}, state=present] ***
2026-03-14 09:20:03.344573 | controller | Saturday 14 March 2026  09:20:03 +0000 (0:00:00.032)       0:00:55.037 ********
2026-03-14 09:20:03.344591 | controller | Saturday 14 March 2026  09:20:03 +0000 (0:00:00.032)       0:00:55.036 ********
2026-03-14 09:20:03.344640 | controller | skipping: [controller]
2026-03-14 09:20:03.434112 | controller |
2026-03-14 09:20:03.434155 | controller | TASK [install_yamls : Compute the cifmw_install_yamls_vars final value _install_yamls_override_vars={{
2026-03-14 09:20:03.434164 | controller |   _install_yamls_override_vars | default({}) |
2026-03-14 09:20:03.434171 | controller |   combine(item, recursive=True)
2026-03-14 09:20:03.434178 | controller | }}] ***
2026-03-14 09:20:03.434193 | controller | Saturday 14 March 2026  09:20:03 +0000 (0:00:00.035)       0:00:55.073 ********
2026-03-14 09:20:03.434200 | controller | Saturday 14 March 2026  09:20:03 +0000 (0:00:00.035)       0:00:55.072 ********
2026-03-14 09:20:03.434213 | controller | ok: [controller] => (item={'BMO_SETUP': False})
2026-03-14 09:20:03.478331 | controller |
2026-03-14 09:20:03.478396 | controller | TASK [install_yamls : Set environment override cifmw_install_yamls_environment fact cifmw_install_yamls_environment={{
2026-03-14 09:20:03.478404 | controller |   _install_yamls_override_vars.keys() |
2026-03-14 09:20:03.478412 | controller |   map('upper') |
2026-03-14 09:20:03.478418 | controller |   zip(_install_yamls_override_vars.values()) |
2026-03-14 09:20:03.478425 | controller |   items2dict(key_name=0, value_name=1) |
2026-03-14 09:20:03.478432 | controller |   combine({
2026-03-14 09:20:03.478439 | controller |     'OUT': cifmw_install_yamls_manifests_dir,
2026-03-14 09:20:03.478445 | controller |     'OUTPUT_DIR': cifmw_install_yamls_edpm_dir,
2026-03-14 09:20:03.478452 | controller |     'CHECKOUT_FROM_OPENSTACK_REF': cifmw_install_yamls_checkout_openstack_ref,
2026-03-14 09:20:03.478459 | controller |     'OPENSTACK_K8S_BRANCH': (zuul is defined and not zuul.branch |regex_search('master|antelope|rhos')) | ternary(zuul.branch, 'main')
2026-03-14 09:20:03.478466 | controller |   }) |
2026-03-14 09:20:03.478473 | controller |   combine(install_yamls_operators_repos)
2026-03-14 09:20:03.478487 | controller | }}, cacheable=True] ***
2026-03-14 09:20:03.478494 | controller | Saturday 14 March 2026  09:20:03 +0000 (0:00:00.089)       0:00:55.163 ********
2026-03-14 09:20:03.478501 | controller | Saturday 14 March 2026  09:20:03 +0000 (0:00:00.089)       0:00:55.162 ********
2026-03-14 09:20:03.478517 | controller | ok: [controller]
2026-03-14 09:20:04.018144 | controller |
2026-03-14 09:20:04.018193 | controller | TASK [install_yamls : Get environment structure base_path={{ cifmw_install_yamls_repo }}] ***
2026-03-14 09:20:04.018200 | controller | Saturday 14 March 2026  09:20:03 +0000 (0:00:00.044)       0:00:55.208 ********
2026-03-14 09:20:04.018205 | controller | Saturday 14 March 2026  09:20:03 +0000 (0:00:00.044)       0:00:55.206 ********
2026-03-14 09:20:04.018215 | controller | ok: [controller]
2026-03-14 09:20:04.229950 | controller |
2026-03-14 09:20:04.229986 | controller | TASK [install_yamls : Ensure Output directory exists path={{ cifmw_install_yamls_out_dir }}, state=directory, mode=0755] ***
2026-03-14 09:20:04.229993 | controller | Saturday 14 March 2026  09:20:04 +0000 (0:00:00.539)       0:00:55.747 ********
2026-03-14 09:20:04.229999 | controller | Saturday 14 March 2026  09:20:04 +0000 (0:00:00.539)       0:00:55.746 ********
2026-03-14 09:20:04.230009 | controller | ok: [controller]
2026-03-14 09:20:04.272141 | controller |
2026-03-14 09:20:04.272174 | controller | TASK [install_yamls : Ensure user cifmw_install_yamls_vars contains existing Makefile variables that=_cifmw_install_yamls_unmatched_vars | length == 0, msg=cifmw_install_yamls_vars contains a variable that is not defined in install_yamls Makefile nor cifmw_install_yamls_whitelisted_vars: {{ _cifmw_install_yamls_unmatched_vars | join(', ')}}, quiet=True] ***
2026-03-14 09:20:04.272181 | controller | Saturday 14 March 2026  09:20:04 +0000 (0:00:00.210)       0:00:55.958 ********
2026-03-14 09:20:04.272186 | controller | Saturday 14 March 2026  09:20:04 +0000 (0:00:00.210)       0:00:55.957 ********
2026-03-14 09:20:04.272195 | controller | ok: [controller]
2026-03-14 09:20:04.754613 | controller |
2026-03-14 09:20:04.754645 | controller | TASK [install_yamls : Generate /home/zuul/ci-framework-data/artifacts/install_yamls.sh dest={{ cifmw_install_yamls_out_dir }}/{{ cifmw_install_yamls_envfile }}, content={% for k,v in cifmw_install_yamls_environment.items() %}
2026-03-14 09:20:04.754652 | controller | export {{ k }}={{ v }}
2026-03-14 09:20:04.754657 | controller | {% endfor %}, mode=0644] ***
2026-03-14 09:20:04.754665 | controller | Saturday 14 March 2026  09:20:04 +0000 (0:00:00.043)       0:00:56.001 ********
2026-03-14 09:20:04.754670 | controller | Saturday 14 March 2026  09:20:04 +0000 (0:00:00.043)       0:00:56.000 ********
2026-03-14 09:20:04.754680 | controller | changed: [controller]
2026-03-14 09:20:04.780407 | controller |
2026-03-14 09:20:04.780468 | controller | TASK [install_yamls : Set install_yamls default values cifmw_install_yamls_defaults={{ get_makefiles_env_output.makefiles_values  | combine(cifmw_install_yamls_environment) }}, cacheable=True] ***
2026-03-14 09:20:04.780485 | controller | Saturday 14 March 2026  09:20:04 +0000 (0:00:00.482)       0:00:56.484 ********
2026-03-14 09:20:04.780490 | controller | Saturday 14 March 2026  09:20:04 +0000 (0:00:00.482)       0:00:56.482 ********
2026-03-14 09:20:04.780502 | controller | ok: [controller]
2026-03-14 09:20:04.801886 | controller |
2026-03-14 09:20:04.801936 | controller | TASK [install_yamls : Show the env structure var=cifmw_install_yamls_environment] ***
2026-03-14 09:20:04.801943 | controller | Saturday 14 March 2026  09:20:04 +0000 (0:00:00.025)       0:00:56.510 ********
2026-03-14 09:20:04.801948 | controller | Saturday 14 March 2026  09:20:04 +0000 (0:00:00.025)       0:00:56.508 ********
2026-03-14 09:20:04.801959 | controller | ok: [controller] =>
2026-03-14 09:20:04.825727 | controller |     cifmw_install_yamls_environment:
2026-03-14 09:20:04.825773 | controller |       BMO_SETUP: false
2026-03-14 09:20:04.825779 | controller |       CHECKOUT_FROM_OPENSTACK_REF: 'true'
2026-03-14 09:20:04.825784 | controller |       OPENSTACK_K8S_BRANCH: main
2026-03-14 09:20:04.825789 | controller |       OUT: /home/zuul/ci-framework-data/artifacts/manifests
2026-03-14 09:20:04.825797 | controller |       OUTPUT_DIR: /home/zuul/ci-framework-data/artifacts/edpm
2026-03-14 09:20:04.825802 | controller |       WATCHER_BRANCH: ''
2026-03-14 09:20:04.825806 | controller |       WATCHER_REPO: /home/zuul/src/github.com/openstack-k8s-operators/watcher-operator
2026-03-14 09:20:04.825811 | controller |
2026-03-14 09:20:04.825816 | controller | TASK [install_yamls : Show the env structure defaults var=cifmw_install_yamls_defaults] ***
2026-03-14 09:20:04.825820 | controller | Saturday 14 March 2026  09:20:04 +0000 (0:00:00.021)       0:00:56.531 ********
2026-03-14 09:20:04.825825 | controller | Saturday 14 March 2026  09:20:04 +0000 (0:00:00.021)       0:00:56.530 ********
2026-03-14 09:20:04.825836 | controller | ok: [controller] =>
2026-03-14 09:20:04.825841 | controller |     cifmw_install_yamls_defaults:
2026-03-14 09:20:04.825846 | controller |       ADOPTED_EXTERNAL_NETWORK: 172.21.1.0/24
2026-03-14 09:20:04.825850 | controller |       ADOPTED_INTERNALAPI_NETWORK: 172.17.1.0/24
2026-03-14 09:20:04.825855 | controller |       ADOPTED_STORAGEMGMT_NETWORK: 172.20.1.0/24
2026-03-14 09:20:04.825859 | controller |       ADOPTED_STORAGE_NETWORK: 172.18.1.0/24
2026-03-14 09:20:04.825863 | controller |       ADOPTED_TENANT_NETWORK: 172.9.1.0/24
2026-03-14 09:20:04.825868 | controller |       BARBICAN: config/samples/barbican_v1beta1_barbican.yaml
2026-03-14 09:20:04.825872 | controller |       BARBICAN_BRANCH: main
2026-03-14 09:20:04.825876 | controller |       BARBICAN_COMMIT_HASH: ''
2026-03-14 09:20:04.825881 | controller |       BARBICAN_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/barbican-operator/config/samples/barbican_v1beta1_barbican.yaml
2026-03-14 09:20:04.825886 | controller |       BARBICAN_DEPL_IMG: unused
2026-03-14 09:20:04.825890 | controller |       BARBICAN_IMG: quay.io/openstack-k8s-operators/barbican-operator-index:latest
2026-03-14 09:20:04.825895 | controller |       BARBICAN_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/barbican-operator/kuttl-test.yaml
2026-03-14 09:20:04.825899 | controller |       BARBICAN_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/barbican-operator/test/kuttl/tests
2026-03-14 09:20:04.825904 | controller |       BARBICAN_KUTTL_NAMESPACE: barbican-kuttl-tests
2026-03-14 09:20:04.825908 | controller |       BARBICAN_REPO: https://github.com/openstack-k8s-operators/barbican-operator.git
2026-03-14 09:20:04.825912 | controller |       BARBICAN_SERVICE_ENABLED: 'true'
2026-03-14 09:20:04.825917 | controller |       BARBICAN_SIMPLE_CRYPTO_ENCRYPTION_KEY: sEFmdFjDUqRM2VemYslV5yGNWjokioJXsg8Nrlc3drU=
2026-03-14 09:20:04.825921 | controller |       BAREMETAL_BRANCH: main
2026-03-14 09:20:04.825926 | controller |       BAREMETAL_COMMIT_HASH: ''
2026-03-14 09:20:04.825932 | controller |       BAREMETAL_IMG: quay.io/openstack-k8s-operators/openstack-baremetal-operator-index:latest
2026-03-14 09:20:04.825937 | controller |       BAREMETAL_OS_CONTAINER_IMG: ''
2026-03-14 09:20:04.825941 | controller |       BAREMETAL_OS_IMG: ''
2026-03-14 09:20:04.825947 | controller |       BAREMETAL_OS_IMG_TYPE: ''
2026-03-14 09:20:04.825952 | controller |       BAREMETAL_REPO: https://github.com/openstack-k8s-operators/openstack-baremetal-operator.git
2026-03-14 09:20:04.825956 | controller |       BAREMETAL_TIMEOUT: 20m
2026-03-14 09:20:04.825961 | controller |       BASH_IMG: quay.io/openstack-k8s-operators/bash:latest
2026-03-14 09:20:04.825965 | controller |       BGP_ASN: '64999'
2026-03-14 09:20:04.825977 | controller |       BGP_LEAF_1: 100.65.4.1
2026-03-14 09:20:04.825981 | controller |       BGP_LEAF_2: 100.64.4.1
2026-03-14 09:20:04.825986 | controller |       BGP_OVN_ROUTING: 'false'
2026-03-14 09:20:04.825990 | controller |       BGP_PEER_ASN: '64999'
2026-03-14 09:20:04.825994 | controller |       BGP_SOURCE_IP: 172.30.4.2
2026-03-14 09:20:04.825999 | controller |       BGP_SOURCE_IP6: f00d:f00d:f00d:f00d:f00d:f00d:f00d:42
2026-03-14 09:20:04.826003 | controller |       BMAAS_BRIDGE_IPV4_PREFIX: 172.20.1.2/24
2026-03-14 09:20:04.826008 | controller |       BMAAS_BRIDGE_IPV6_PREFIX: fd00:bbbb::2/64
2026-03-14 09:20:04.826012 | controller |       BMAAS_INSTANCE_DISK_SIZE: '20'
2026-03-14 09:20:04.826016 | controller |       BMAAS_INSTANCE_MEMORY: '4096'
2026-03-14 09:20:04.826020 | controller |       BMAAS_INSTANCE_NAME_PREFIX: crc-bmaas
2026-03-14 09:20:04.826025 | controller |       BMAAS_INSTANCE_NET_MODEL: virtio
2026-03-14 09:20:04.826030 | controller |       BMAAS_INSTANCE_OS_VARIANT: centos-stream9
2026-03-14 09:20:04.826034 | controller |       BMAAS_INSTANCE_VCPUS: '2'
2026-03-14 09:20:04.826038 | controller |       BMAAS_INSTANCE_VIRT_TYPE: kvm
2026-03-14 09:20:04.826043 | controller |       BMAAS_IPV4: 'true'
2026-03-14 09:20:04.826047 | controller |       BMAAS_IPV6: 'false'
2026-03-14 09:20:04.826051 | controller |       BMAAS_LIBVIRT_USER: sushyemu
2026-03-14 09:20:04.826056 | controller |       BMAAS_METALLB_ADDRESS_POOL: 172.20.1.64/26
2026-03-14 09:20:04.826060 | controller |       BMAAS_METALLB_POOL_NAME: baremetal
2026-03-14 09:20:04.826064 | controller |       BMAAS_NETWORK_IPV4_PREFIX: 172.20.1.1/24
2026-03-14 09:20:04.826069 | controller |       BMAAS_NETWORK_IPV6_PREFIX: fd00:bbbb::1/64
2026-03-14 09:20:04.826073 | controller |       BMAAS_NETWORK_NAME: crc-bmaas
2026-03-14 09:20:04.826077 | controller |       BMAAS_NODE_COUNT: '1'
2026-03-14 09:20:04.826081 | controller |       BMAAS_OCP_INSTANCE_NAME: crc
2026-03-14 09:20:04.826086 | controller |       BMAAS_REDFISH_PASSWORD: password
2026-03-14 09:20:04.826090 | controller |       BMAAS_REDFISH_USERNAME: admin
2026-03-14 09:20:04.826094 | controller |       BMAAS_ROUTE_LIBVIRT_NETWORKS: crc-bmaas,crc,default
2026-03-14 09:20:04.826099 | controller |       BMAAS_SUSHY_EMULATOR_DRIVER: libvirt
2026-03-14 09:20:04.826103 | controller |       BMAAS_SUSHY_EMULATOR_IMAGE: quay.io/metal3-io/sushy-tools:latest
2026-03-14 09:20:04.826107 | controller |       BMAAS_SUSHY_EMULATOR_NAMESPACE: sushy-emulator
2026-03-14 09:20:04.826112 | controller |       BMAAS_SUSHY_EMULATOR_OS_CLIENT_CONFIG_FILE: /etc/openstack/clouds.yaml
2026-03-14 09:20:04.826116 | controller |       BMAAS_SUSHY_EMULATOR_OS_CLOUD: openstack
2026-03-14 09:20:04.826120 | controller |       BMH_NAMESPACE: openstack
2026-03-14 09:20:04.826125 | controller |       BMO_BRANCH: release-0.9
2026-03-14 09:20:04.826129 | controller |       BMO_CLEANUP: 'true'
2026-03-14 09:20:04.826133 | controller |       BMO_COMMIT_HASH: ''
2026-03-14 09:20:04.826138 | controller |       BMO_IPA_BASEURI: https://tarballs.opendev.org/openstack/ironic-python-agent/dib
2026-03-14 09:20:04.826142 | controller |       BMO_IPA_BRANCH: stable/2024.1
2026-03-14 09:20:04.826146 | controller |       BMO_IPA_FLAVOR: centos9
2026-03-14 09:20:04.826151 | controller |       BMO_IPA_INSECURE: 'false'
2026-03-14 09:20:04.826155 | controller |       BMO_IRONIC_HOST: 192.168.122.10
2026-03-14 09:20:04.826159 | controller |       BMO_PROVISIONING_INTERFACE: ''
2026-03-14 09:20:04.826164 | controller |       BMO_REPO: https://github.com/metal3-io/baremetal-operator
2026-03-14 09:20:04.826168 | controller |       BMO_SETUP: false
2026-03-14 09:20:04.826172 | controller |       BMO_SETUP_ROUTE_REPLACE: 'true'
2026-03-14 09:20:04.826176 | controller |       BM_CTLPLANE_INTERFACE: enp1s0
2026-03-14 09:20:04.826181 | controller |       BM_INSTANCE_MEMORY: '8192'
2026-03-14 09:20:04.826185 | controller |       BM_INSTANCE_NAME_PREFIX: edpm-compute-baremetal
2026-03-14 09:20:04.826189 | controller |       BM_INSTANCE_NAME_SUFFIX: '0'
2026-03-14 09:20:04.826194 | controller |       BM_NETWORK_NAME: default
2026-03-14 09:20:04.826198 | controller |       BM_NODE_COUNT: '1'
2026-03-14 09:20:04.826202 | controller |       BM_ROOT_PASSWORD: ''
2026-03-14 09:20:04.826207 | controller |       BM_ROOT_PASSWORD_SECRET: ''
2026-03-14 09:20:04.826211 | controller |       CEILOMETER_CENTRAL_DEPL_IMG: unused
2026-03-14 09:20:04.826215 | controller |       CEILOMETER_NOTIFICATION_DEPL_IMG: unused
2026-03-14 09:20:04.826219 | controller |       CEPH_BRANCH: release-1.15
2026-03-14 09:20:04.826224 | controller |       CEPH_CLIENT: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/toolbox.yaml
2026-03-14 09:20:04.826232 | controller |       CEPH_COMMON: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/common.yaml
2026-03-14 09:20:04.826238 | controller |       CEPH_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/cluster-test.yaml
2026-03-14 09:20:04.826243 | controller |       CEPH_CRDS: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/crds.yaml
2026-03-14 09:20:04.826247 | controller |       CEPH_IMG: quay.io/ceph/demo:latest-squid
2026-03-14 09:20:04.826252 | controller |       CEPH_OP: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/operator-openshift.yaml
2026-03-14 09:20:04.826256 | controller |       CEPH_REPO: https://github.com/rook/rook.git
2026-03-14 09:20:04.826260 | controller |       CERTMANAGER_TIMEOUT: 300s
2026-03-14 09:20:04.826265 | controller |       CHECKOUT_FROM_OPENSTACK_REF: 'true'
2026-03-14 09:20:04.826269 | controller |       CINDER: config/samples/cinder_v1beta1_cinder.yaml
2026-03-14 09:20:04.826273 | controller |       CINDERAPI_DEPL_IMG: unused
2026-03-14 09:20:04.826278 | controller |       CINDERBKP_DEPL_IMG: unused
2026-03-14 09:20:04.826282 | controller |       CINDERSCH_DEPL_IMG: unused
2026-03-14 09:20:04.826286 | controller |       CINDERVOL_DEPL_IMG: unused
2026-03-14 09:20:04.826291 | controller |       CINDER_BRANCH: main
2026-03-14 09:20:04.826295 | controller |       CINDER_COMMIT_HASH: ''
2026-03-14 09:20:04.826299 | controller |       CINDER_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/cinder-operator/config/samples/cinder_v1beta1_cinder.yaml
2026-03-14 09:20:04.826304 | controller |       CINDER_IMG: quay.io/openstack-k8s-operators/cinder-operator-index:latest
2026-03-14 09:20:04.826308 | controller |       CINDER_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/cinder-operator/kuttl-test.yaml
2026-03-14 09:20:04.826312 | controller |       CINDER_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/cinder-operator/test/kuttl/tests
2026-03-14 09:20:04.826316 | controller |       CINDER_KUTTL_NAMESPACE: cinder-kuttl-tests
2026-03-14 09:20:04.826321 | controller |       CINDER_REPO: https://github.com/openstack-k8s-operators/cinder-operator.git
2026-03-14 09:20:04.826325 | controller |       CLEANUP_DIR_CMD: rm -Rf
2026-03-14 09:20:04.826329 | controller |       CRC_BGP_NIC_1_MAC: '52:54:00:11:11:11'
2026-03-14 09:20:04.826334 | controller |       CRC_BGP_NIC_2_MAC: '52:54:00:11:11:12'
2026-03-14 09:20:04.826341 | controller |       CRC_HTTPS_PROXY: ''
2026-03-14 09:20:04.826346 | controller |       CRC_HTTP_PROXY: ''
2026-03-14 09:20:04.826350 | controller |       CRC_STORAGE_NAMESPACE: crc-storage
2026-03-14 09:20:04.826355 | controller |       CRC_STORAGE_RETRIES: '3'
2026-03-14 09:20:04.826359 | controller |       CRC_URL: '''https://developers.redhat.com/content-gateway/rest/mirror/pub/openshift-v4/clients/crc/latest/crc-linux-amd64.tar.xz'''
2026-03-14 09:20:04.826363 | controller |       CRC_VERSION: latest
2026-03-14 09:20:04.826368 | controller |       DATAPLANE_ANSIBLE_SECRET: dataplane-ansible-ssh-private-key-secret
2026-03-14 09:20:04.826372 | controller |       DATAPLANE_ANSIBLE_USER: ''
2026-03-14 09:20:04.826376 | controller |       DATAPLANE_COMPUTE_IP: 192.168.122.100
2026-03-14 09:20:04.826381 | controller |       DATAPLANE_CONTAINER_PREFIX: openstack
2026-03-14 09:20:04.826385 | controller |       DATAPLANE_CONTAINER_TAG: current-podified
2026-03-14 09:20:04.826389 | controller |       DATAPLANE_CUSTOM_SERVICE_RUNNER_IMG: quay.io/openstack-k8s-operators/openstack-ansibleee-runner:latest
2026-03-14 09:20:04.826394 | controller |       DATAPLANE_DEFAULT_GW: 192.168.122.1
2026-03-14 09:20:04.826398 | controller |       DATAPLANE_EXTRA_NOVA_CONFIG_FILE: /dev/null
2026-03-14 09:20:04.826402 | controller |       DATAPLANE_GROWVOLS_ARGS: /=8GB /tmp=1GB /home=1GB /var=100%
2026-03-14 09:20:04.826406 | controller |       DATAPLANE_KUSTOMIZE_SCENARIO: preprovisioned
2026-03-14 09:20:04.826411 | controller |       DATAPLANE_NETWORKER_IP: 192.168.122.200
2026-03-14 09:20:04.826415 | controller |       DATAPLANE_NETWORK_INTERFACE_NAME: eth0
2026-03-14 09:20:04.826419 | controller |       DATAPLANE_NOVA_NFS_PATH: ''
2026-03-14 09:20:04.826424 | controller |       DATAPLANE_NTP_SERVER: pool.ntp.org
2026-03-14 09:20:04.826428 | controller |       DATAPLANE_PLAYBOOK: osp.edpm.download_cache
2026-03-14 09:20:04.826432 | controller |       DATAPLANE_REGISTRY_URL: quay.io/podified-antelope-centos9
2026-03-14 09:20:04.826437 | controller |       DATAPLANE_RUNNER_IMG: ''
2026-03-14 09:20:04.826441 | controller |       DATAPLANE_SERVER_ROLE: compute
2026-03-14 09:20:04.826447 | controller |       DATAPLANE_SSHD_ALLOWED_RANGES: '[''192.168.122.0/24'']'
2026-03-14 09:20:04.826452 | controller |       DATAPLANE_TIMEOUT: 30m
2026-03-14 09:20:04.826456 | controller |       DATAPLANE_TLS_ENABLED: 'true'
2026-03-14 09:20:04.826460 | controller |       DATAPLANE_TOTAL_NETWORKER_NODES: '1'
2026-03-14 09:20:04.826465 | controller |       DATAPLANE_TOTAL_NODES: '1'
2026-03-14 09:20:04.826469 | controller |       DBSERVICE: galera
2026-03-14 09:20:04.826473 | controller |       DESIGNATE: config/samples/designate_v1beta1_designate.yaml
2026-03-14 09:20:04.826477 | controller |       DESIGNATE_BRANCH: main
2026-03-14 09:20:04.826482 | controller |       DESIGNATE_COMMIT_HASH: ''
2026-03-14 09:20:04.826486 | controller |       DESIGNATE_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/designate-operator/config/samples/designate_v1beta1_designate.yaml
2026-03-14 09:20:04.826490 | controller |       DESIGNATE_IMG: quay.io/openstack-k8s-operators/designate-operator-index:latest
2026-03-14 09:20:04.826495 | controller |       DESIGNATE_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/designate-operator/kuttl-test.yaml
2026-03-14 09:20:04.826499 | controller |       DESIGNATE_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/designate-operator/test/kuttl/tests
2026-03-14 09:20:04.826503 | controller |       DESIGNATE_KUTTL_NAMESPACE: designate-kuttl-tests
2026-03-14 09:20:04.826508 | controller |       DESIGNATE_REPO: https://github.com/openstack-k8s-operators/designate-operator.git
2026-03-14 09:20:04.826512 | controller |       DNSDATA: config/samples/network_v1beta1_dnsdata.yaml
2026-03-14 09:20:04.826518 | controller |       DNSDATA_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/config/samples/network_v1beta1_dnsdata.yaml
2026-03-14 09:20:04.826522 | controller |       DNSMASQ: config/samples/network_v1beta1_dnsmasq.yaml
2026-03-14 09:20:04.826527 | controller |       DNSMASQ_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/config/samples/network_v1beta1_dnsmasq.yaml
2026-03-14 09:20:04.826531 | controller |       DNS_DEPL_IMG: unused
2026-03-14 09:20:04.826535 | controller |       DNS_DOMAIN: localdomain
2026-03-14 09:20:04.826542 | controller |       DOWNLOAD_TOOLS_SELECTION: all
2026-03-14 09:20:04.826547 | controller |       EDPM_ATTACH_EXTNET: 'true'
2026-03-14 09:20:04.826551 | controller |       EDPM_COMPUTE_ADDITIONAL_HOST_ROUTES: '''[]'''
2026-03-14 09:20:04.826556 | controller |       EDPM_COMPUTE_ADDITIONAL_NETWORKS: '''[]'''
2026-03-14 09:20:04.826560 | controller |       EDPM_COMPUTE_CELLS: '1'
2026-03-14 09:20:04.826564 | controller |       EDPM_COMPUTE_CEPH_ENABLED: 'true'
2026-03-14 09:20:04.826569 | controller |       EDPM_COMPUTE_CEPH_NOVA: 'true'
2026-03-14 09:20:04.826573 | controller |       EDPM_COMPUTE_DHCP_AGENT_ENABLED: 'true'
2026-03-14 09:20:04.826577 | controller |       EDPM_COMPUTE_SRIOV_ENABLED: 'true'
2026-03-14 09:20:04.826581 | controller |       EDPM_COMPUTE_SUFFIX: '0'
2026-03-14 09:20:04.826586 | controller |       EDPM_CONFIGURE_DEFAULT_ROUTE: 'true'
2026-03-14 09:20:04.826590 | controller |       EDPM_CONFIGURE_HUGEPAGES: 'false'
2026-03-14 09:20:04.826610 | controller |       EDPM_CONFIGURE_NETWORKING: 'true'
2026-03-14 09:20:04.826617 | controller |       EDPM_FIRSTBOOT_EXTRA: /tmp/edpm-firstboot-extra
2026-03-14 09:20:04.826621 | controller |       EDPM_NETWORKER_SUFFIX: '0'
2026-03-14 09:20:04.826626 | controller |       EDPM_TOTAL_NETWORKERS: '1'
2026-03-14 09:20:04.826630 | controller |       EDPM_TOTAL_NODES: '1'
2026-03-14 09:20:04.826634 | controller |       GALERA_REPLICAS: ''
2026-03-14 09:20:04.826639 | controller |       GENERATE_SSH_KEYS: 'true'
2026-03-14 09:20:04.826643 | controller |       GIT_CLONE_OPTS: ''
2026-03-14 09:20:04.826647 | controller |       GLANCE: config/samples/glance_v1beta1_glance.yaml
2026-03-14 09:20:04.826652 | controller |       GLANCEAPI_DEPL_IMG: unused
2026-03-14 09:20:04.826656 | controller |       GLANCE_BRANCH: main
2026-03-14 09:20:04.826660 | controller |       GLANCE_COMMIT_HASH: ''
2026-03-14 09:20:04.826665 | controller |       GLANCE_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/glance-operator/config/samples/glance_v1beta1_glance.yaml
2026-03-14 09:20:04.826670 | controller |       GLANCE_IMG: quay.io/openstack-k8s-operators/glance-operator-index:latest
2026-03-14 09:20:04.826674 | controller |       GLANCE_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/glance-operator/kuttl-test.yaml
2026-03-14 09:20:04.826678 | controller |       GLANCE_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/glance-operator/test/kuttl/tests
2026-03-14 09:20:04.826685 | controller |       GLANCE_KUTTL_NAMESPACE: glance-kuttl-tests
2026-03-14 09:20:04.826689 | controller |       GLANCE_REPO: https://github.com/openstack-k8s-operators/glance-operator.git
2026-03-14 09:20:04.826695 | controller |       HEAT: config/samples/heat_v1beta1_heat.yaml
2026-03-14 09:20:04.826700 | controller |       HEATAPI_DEPL_IMG: unused
2026-03-14 09:20:04.826704 | controller |       HEATCFNAPI_DEPL_IMG: unused
2026-03-14 09:20:04.826709 | controller |       HEATENGINE_DEPL_IMG: unused
2026-03-14 09:20:04.826713 | controller |       HEAT_AUTH_ENCRYPTION_KEY: 767c3ed056cbaa3b9dfedb8c6f825bf0
2026-03-14 09:20:04.826717 | controller |       HEAT_BRANCH: main
2026-03-14 09:20:04.826722 | controller |       HEAT_COMMIT_HASH: ''
2026-03-14 09:20:04.826728 | controller |       HEAT_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/heat-operator/config/samples/heat_v1beta1_heat.yaml
2026-03-14 09:20:04.826733 | controller |       HEAT_IMG: quay.io/openstack-k8s-operators/heat-operator-index:latest
2026-03-14 09:20:04.826737 | controller |       HEAT_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/heat-operator/kuttl-test.yaml
2026-03-14 09:20:04.826743 | controller |       HEAT_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/heat-operator/test/kuttl/tests
2026-03-14 09:20:04.826747 | controller |       HEAT_KUTTL_NAMESPACE: heat-kuttl-tests
2026-03-14 09:20:04.826752 | controller |       HEAT_REPO: https://github.com/openstack-k8s-operators/heat-operator.git
2026-03-14 09:20:04.826756 | controller |       HEAT_SERVICE_ENABLED: 'true'
2026-03-14 09:20:04.826760 | controller |       HORIZON: config/samples/horizon_v1beta1_horizon.yaml
2026-03-14 09:20:04.826765 | controller |       HORIZON_BRANCH: main
2026-03-14 09:20:04.826769 | controller |       HORIZON_COMMIT_HASH: ''
2026-03-14 09:20:04.826802 | controller |       HORIZON_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/horizon-operator/config/samples/horizon_v1beta1_horizon.yaml
2026-03-14 09:20:04.826809 | controller |       HORIZON_DEPL_IMG: unused
2026-03-14 09:20:04.826814 | controller |       HORIZON_IMG: quay.io/openstack-k8s-operators/horizon-operator-index:latest
2026-03-14 09:20:04.826819 | controller |       HORIZON_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/horizon-operator/kuttl-test.yaml
2026-03-14 09:20:04.826826 | controller |       HORIZON_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/horizon-operator/test/kuttl/tests
2026-03-14 09:20:04.826831 | controller |       HORIZON_KUTTL_NAMESPACE: horizon-kuttl-tests
2026-03-14 09:20:04.826837 | controller |       HORIZON_REPO: https://github.com/openstack-k8s-operators/horizon-operator.git
2026-03-14 09:20:04.826842 | controller |       INFRA_BRANCH: main
2026-03-14 09:20:04.826848 | controller |       INFRA_COMMIT_HASH: ''
2026-03-14 09:20:04.826853 | controller |       INFRA_IMG: quay.io/openstack-k8s-operators/infra-operator-index:latest
2026-03-14 09:20:04.826859 | controller |       INFRA_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/kuttl-test.yaml
2026-03-14 09:20:04.826864 | controller |       INFRA_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/test/kuttl/tests
2026-03-14 09:20:04.826870 | controller |       INFRA_KUTTL_NAMESPACE: infra-kuttl-tests
2026-03-14 09:20:04.826874 | controller |       INFRA_REPO: https://github.com/openstack-k8s-operators/infra-operator.git
2026-03-14 09:20:04.826879 | controller |       INSTALL_CERT_MANAGER: 'true'
2026-03-14 09:20:04.826883 | controller |       INSTALL_NMSTATE: true || false
2026-03-14 09:20:04.826887 | controller |       INSTALL_NNCP: true || false
2026-03-14 09:20:04.826891 | controller |       INTERNALAPI_HOST_ROUTES: ''
2026-03-14 09:20:04.826895 | controller |       IPV6_LAB_IPV4_NETWORK_IPADDRESS: 172.30.0.1/24
2026-03-14 09:20:04.826900 | controller |       IPV6_LAB_IPV6_NETWORK_IPADDRESS: fd00:abcd:abcd:fc00::1/64
2026-03-14 09:20:04.826904 | controller |       IPV6_LAB_LIBVIRT_STORAGE_POOL: default
2026-03-14 09:20:04.826908 | controller |       IPV6_LAB_MANAGE_FIREWALLD: 'true'
2026-03-14 09:20:04.826912 | controller |       IPV6_LAB_NAT64_HOST_IPV4: 172.30.0.2/24
2026-03-14 09:20:04.826917 | controller |       IPV6_LAB_NAT64_HOST_IPV6: fd00:abcd:abcd:fc00::2/64
2026-03-14 09:20:04.826921 | controller |       IPV6_LAB_NAT64_INSTANCE_NAME: nat64-router
2026-03-14 09:20:04.826925 | controller |       IPV6_LAB_NAT64_IPV6_NETWORK: fd00:abcd:abcd:fc00::/64
2026-03-14 09:20:04.826939 | controller |       IPV6_LAB_NAT64_TAYGA_DYNAMIC_POOL: 192.168.255.0/24
2026-03-14 09:20:04.826943 | controller |       IPV6_LAB_NAT64_TAYGA_IPV4: 192.168.255.1
2026-03-14 09:20:04.826949 | controller |       IPV6_LAB_NAT64_TAYGA_IPV6: fd00:abcd:abcd:fc00::3
2026-03-14 09:20:04.826953 | controller |       IPV6_LAB_NAT64_TAYGA_IPV6_PREFIX: fd00:abcd:abcd:fcff::/96
2026-03-14 09:20:04.826957 | controller |       IPV6_LAB_NAT64_UPDATE_PACKAGES: 'false'
2026-03-14 09:20:04.826961 | controller |       IPV6_LAB_NETWORK_NAME: nat64
2026-03-14 09:20:04.826966 | controller |       IPV6_LAB_SNO_CLUSTER_NETWORK: fd00:abcd:0::/48
2026-03-14 09:20:04.826970 | controller |       IPV6_LAB_SNO_HOST_IP: fd00:abcd:abcd:fc00::11
2026-03-14 09:20:04.826974 | controller |       IPV6_LAB_SNO_HOST_PREFIX: '64'
2026-03-14 09:20:04.826978 | controller |       IPV6_LAB_SNO_INSTANCE_NAME: sno
2026-03-14 09:20:04.826983 | controller |       IPV6_LAB_SNO_MACHINE_NETWORK: fd00:abcd:abcd:fc00::/64
2026-03-14 09:20:04.826987 | controller |       IPV6_LAB_SNO_OCP_MIRROR_URL: https://mirror.openshift.com/pub/openshift-v4/clients/ocp
2026-03-14 09:20:04.826991 | controller |       IPV6_LAB_SNO_OCP_VERSION: latest-4.14
2026-03-14 09:20:04.826995 | controller |       IPV6_LAB_SNO_SERVICE_NETWORK: fd00:abcd:abcd:fc03::/112
2026-03-14 09:20:04.827000 | controller |       IPV6_LAB_SSH_PUB_KEY: /home/zuul/.ssh/id_rsa.pub
2026-03-14 09:20:04.827004 | controller |       IPV6_LAB_WORK_DIR: /home/zuul/.ipv6lab
2026-03-14 09:20:04.827008 | controller |       IRONIC: config/samples/ironic_v1beta1_ironic.yaml
2026-03-14 09:20:04.827012 | controller |       IRONICAPI_DEPL_IMG: unused
2026-03-14 09:20:04.827017 | controller |       IRONICCON_DEPL_IMG: unused
2026-03-14 09:20:04.827021 | controller |       IRONICINS_DEPL_IMG: unused
2026-03-14 09:20:04.827025 | controller |       IRONICNAG_DEPL_IMG: unused
2026-03-14 09:20:04.827031 | controller |       IRONICPXE_DEPL_IMG: unused
2026-03-14 09:20:04.827036 | controller |       IRONIC_BRANCH: main
2026-03-14 09:20:04.827040 | controller |       IRONIC_COMMIT_HASH: ''
2026-03-14 09:20:04.827044 | controller |       IRONIC_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ironic-operator/config/samples/ironic_v1beta1_ironic.yaml
2026-03-14 09:20:04.827049 | controller |       IRONIC_IMAGE: quay.io/metal3-io/ironic
2026-03-14 09:20:04.827053 | controller |       IRONIC_IMAGE_TAG: release-24.1
2026-03-14 09:20:04.827057 | controller |       IRONIC_IMG: quay.io/openstack-k8s-operators/ironic-operator-index:latest
2026-03-14 09:20:04.827062 | controller |       IRONIC_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/ironic-operator/kuttl-test.yaml
2026-03-14 09:20:04.827066 | controller |       IRONIC_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ironic-operator/test/kuttl/tests
2026-03-14 09:20:04.827070 | controller |       IRONIC_KUTTL_NAMESPACE: ironic-kuttl-tests
2026-03-14 09:20:04.827074 | controller |       IRONIC_REPO: https://github.com/openstack-k8s-operators/ironic-operator.git
2026-03-14 09:20:04.827078 | controller |       KEYSTONEAPI: config/samples/keystone_v1beta1_keystoneapi.yaml
2026-03-14 09:20:04.827083 | controller |       KEYSTONEAPI_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/keystone-operator/config/samples/keystone_v1beta1_keystoneapi.yaml
2026-03-14 09:20:04.827088 | controller |       KEYSTONEAPI_DEPL_IMG: unused
2026-03-14 09:20:04.827093 | controller |       KEYSTONE_BRANCH: main
2026-03-14 09:20:04.827097 | controller |       KEYSTONE_COMMIT_HASH: ''
2026-03-14 09:20:04.827101 | controller |       KEYSTONE_FEDERATION_CLIENT_SECRET: COX8bmlKAWn56XCGMrKQJj7dgHNAOl6f
2026-03-14 09:20:04.827105 | controller |       KEYSTONE_FEDERATION_CRYPTO_PASSPHRASE: openstack
2026-03-14 09:20:04.827110 | controller |       KEYSTONE_IMG: quay.io/openstack-k8s-operators/keystone-operator-index:latest
2026-03-14 09:20:04.827114 | controller |       KEYSTONE_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/keystone-operator/kuttl-test.yaml
2026-03-14 09:20:04.827118 | controller |       KEYSTONE_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/keystone-operator/test/kuttl/tests
2026-03-14 09:20:04.827123 | controller |       KEYSTONE_KUTTL_NAMESPACE: keystone-kuttl-tests
2026-03-14 09:20:04.827127 | controller |       KEYSTONE_REPO: https://github.com/openstack-k8s-operators/keystone-operator.git
2026-03-14 09:20:04.827131 | controller |       KUBEADMIN_PWD: '12345678'
2026-03-14 09:20:04.827136 | controller |       LIBVIRT_SECRET: libvirt-secret
2026-03-14 09:20:04.827142 | controller |       LOKI_DEPLOY_MODE: openshift-network
2026-03-14 09:20:04.827146 | controller |       LOKI_DEPLOY_NAMESPACE: netobserv
2026-03-14 09:20:04.827150 | controller |       LOKI_DEPLOY_SIZE: 1x.demo
2026-03-14 09:20:04.827155 | controller |       LOKI_NAMESPACE: openshift-operators-redhat
2026-03-14 09:20:04.827159 | controller |       LOKI_OPERATOR_GROUP: openshift-operators-redhat-loki
2026-03-14 09:20:04.827163 | controller |       LOKI_SUBSCRIPTION: loki-operator
2026-03-14 09:20:04.827167 | controller |       LVMS_CR: '1'
2026-03-14 09:20:04.827172 | controller |       MANILA: config/samples/manila_v1beta1_manila.yaml
2026-03-14 09:20:04.827176 | controller |       MANILAAPI_DEPL_IMG: unused
2026-03-14 09:20:04.827180 | controller |       MANILASCH_DEPL_IMG: unused
2026-03-14 09:20:04.827185 | controller |       MANILASHARE_DEPL_IMG: unused
2026-03-14 09:20:04.827189 | controller |       MANILA_BRANCH: main
2026-03-14 09:20:04.827193 | controller |       MANILA_COMMIT_HASH: ''
2026-03-14 09:20:04.827197 | controller |       MANILA_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/manila-operator/config/samples/manila_v1beta1_manila.yaml
2026-03-14 09:20:04.827202 | controller |       MANILA_IMG: quay.io/openstack-k8s-operators/manila-operator-index:latest
2026-03-14 09:20:04.827206 | controller |       MANILA_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/manila-operator/kuttl-test.yaml
2026-03-14 09:20:04.827210 | controller |       MANILA_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/manila-operator/test/kuttl/tests
2026-03-14 09:20:04.827214 | controller |       MANILA_KUTTL_NAMESPACE: manila-kuttl-tests
2026-03-14 09:20:04.827219 | controller |       MANILA_REPO: https://github.com/openstack-k8s-operators/manila-operator.git
2026-03-14 09:20:04.827223 | controller |       MANILA_SERVICE_ENABLED: 'true'
2026-03-14 09:20:04.827227 | controller |       MARIADB: config/samples/mariadb_v1beta1_galera.yaml
2026-03-14 09:20:04.827231 | controller |       MARIADB_BRANCH: main
2026-03-14 09:20:04.827236 | controller |       MARIADB_CHAINSAW_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/test/chainsaw/config.yaml
2026-03-14 09:20:04.827243 | controller |       MARIADB_CHAINSAW_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/test/chainsaw/tests
2026-03-14 09:20:04.827248 | controller |       MARIADB_CHAINSAW_NAMESPACE: mariadb-chainsaw-tests
2026-03-14 09:20:04.827252 | controller |       MARIADB_COMMIT_HASH: ''
2026-03-14 09:20:04.827256 | controller |       MARIADB_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/config/samples/mariadb_v1beta1_galera.yaml
2026-03-14 09:20:04.827260 | controller |       MARIADB_DEPL_IMG: unused
2026-03-14 09:20:04.827265 | controller |       MARIADB_IMG: quay.io/openstack-k8s-operators/mariadb-operator-index:latest
2026-03-14 09:20:04.827269 | controller |       MARIADB_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/kuttl-test.yaml
2026-03-14 09:20:04.827273 | controller |       MARIADB_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/test/kuttl/tests
2026-03-14 09:20:04.827278 | controller |       MARIADB_KUTTL_NAMESPACE: mariadb-kuttl-tests
2026-03-14 09:20:04.827282 | controller |       MARIADB_REPO: https://github.com/openstack-k8s-operators/mariadb-operator.git
2026-03-14 09:20:04.827286 | controller |       MEMCACHED: config/samples/memcached_v1beta1_memcached.yaml
2026-03-14 09:20:04.827291 | controller |       MEMCACHED_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/config/samples/memcached_v1beta1_memcached.yaml
2026-03-14 09:20:04.827295 | controller |       MEMCACHED_DEPL_IMG: unused
2026-03-14 09:20:04.827301 | controller |       METADATA_SHARED_SECRET: '1234567842'
2026-03-14 09:20:04.827306 | controller |       METALLB_IPV6_POOL: fd00:aaaa::80-fd00:aaaa::90
2026-03-14 09:20:04.827310 | controller |       METALLB_POOL: 192.168.122.80-192.168.122.90
2026-03-14 09:20:04.827316 | controller |       MICROSHIFT: '0'
2026-03-14 09:20:04.827320 | controller |       MIRROR_INSECURE: 'true'
2026-03-14 09:20:04.827324 | controller |       MIRROR_NAMESPACE: openstack-mirror
2026-03-14 09:20:04.827328 | controller |       MIRROR_REGISTRY_HOST: default-route-openshift-image-registry.apps-crc.testing
2026-03-14 09:20:04.827333 | controller |       NAMESPACE: openstack
2026-03-14 09:20:04.827337 | controller |       NETCONFIG: config/samples/network_v1beta1_netconfig.yaml
2026-03-14 09:20:04.827344 | controller |       NETCONFIG_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/config/samples/network_v1beta1_netconfig.yaml
2026-03-14 09:20:04.827349 | controller |       NETCONFIG_DEPL_IMG: unused
2026-03-14 09:20:04.827353 | controller |       NETOBSERV_DEPLOY_NAMESPACE: netobserv
2026-03-14 09:20:04.827357 | controller |       NETOBSERV_NAMESPACE: openshift-netobserv-operator
2026-03-14 09:20:04.827362 | controller |       NETOBSERV_OPERATOR_GROUP: openshift-netobserv-operator-net
2026-03-14 09:20:04.827366 | controller |       NETOBSERV_SUBSCRIPTION: netobserv-operator
2026-03-14 09:20:04.827370 | controller |       NETWORK_BGP: 'false'
2026-03-14 09:20:04.827374 | controller |       NETWORK_DESIGNATE_ADDRESS_PREFIX: 172.28.0
2026-03-14 09:20:04.827379 | controller |       NETWORK_DESIGNATE_EXT_ADDRESS_PREFIX: 172.50.0
2026-03-14 09:20:04.827383 | controller |       NETWORK_INTERNALAPI_ADDRESS_PREFIX: 172.17.0
2026-03-14 09:20:04.827387 | controller |       NETWORK_ISOLATION: 'true'
2026-03-14 09:20:04.827392 | controller |       NETWORK_ISOLATION_INSTANCE_NAME: crc
2026-03-14 09:20:04.827396 | controller |       NETWORK_ISOLATION_IPV4: 'true'
2026-03-14 09:20:04.827400 | controller |       NETWORK_ISOLATION_IPV4_ADDRESS: 172.16.1.1/24
2026-03-14 09:20:04.827404 | controller |       NETWORK_ISOLATION_IPV4_NAT: 'true'
2026-03-14 09:20:04.827409 | controller |       NETWORK_ISOLATION_IPV6: 'false'
2026-03-14 09:20:04.827413 | controller |       NETWORK_ISOLATION_IPV6_ADDRESS: fd00:aaaa::1/64
2026-03-14 09:20:04.827417 | controller |       NETWORK_ISOLATION_IP_ADDRESS: 192.168.122.10
2026-03-14 09:20:04.827421 | controller |       NETWORK_ISOLATION_MAC: '52:54:00:11:11:10'
2026-03-14 09:20:04.827426 | controller |       NETWORK_ISOLATION_NETWORK_NAME: net-iso
2026-03-14 09:20:04.827430 | controller |       NETWORK_ISOLATION_NET_NAME: default
2026-03-14 09:20:04.827434 | controller |       NETWORK_ISOLATION_USE_DEFAULT_NETWORK: 'true'
2026-03-14 09:20:04.827439 | controller |       NETWORK_MTU: '1500'
2026-03-14 09:20:04.827443 | controller |       NETWORK_STORAGEMGMT_ADDRESS_PREFIX: 172.20.0
2026-03-14 09:20:04.827447 | controller |       NETWORK_STORAGE_ADDRESS_PREFIX: 172.18.0
2026-03-14 09:20:04.827452 | controller |       NETWORK_STORAGE_MACVLAN: ''
2026-03-14 09:20:04.827456 | controller |       NETWORK_TENANT_ADDRESS_PREFIX: 172.19.0
2026-03-14 09:20:04.827460 | controller |       NETWORK_VLAN_START: '20'
2026-03-14 09:20:04.827464 | controller |       NETWORK_VLAN_STEP: '1'
2026-03-14 09:20:04.827469 | controller |       NEUTRONAPI: config/samples/neutron_v1beta1_neutronapi.yaml
2026-03-14 09:20:04.827473 | controller |       NEUTRONAPI_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/neutron-operator/config/samples/neutron_v1beta1_neutronapi.yaml
2026-03-14 09:20:04.827477 | controller |       NEUTRONAPI_DEPL_IMG: unused
2026-03-14 09:20:04.827482 | controller |       NEUTRON_BRANCH: main
2026-03-14 09:20:04.827486 | controller |       NEUTRON_COMMIT_HASH: ''
2026-03-14 09:20:04.827490 | controller |       NEUTRON_IMG: quay.io/openstack-k8s-operators/neutron-operator-index:latest
2026-03-14 09:20:04.827495 | controller |       NEUTRON_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/neutron-operator/kuttl-test.yaml
2026-03-14 09:20:04.827499 | controller |       NEUTRON_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/neutron-operator/test/kuttl/tests
2026-03-14 09:20:04.827503 | controller |       NEUTRON_KUTTL_NAMESPACE: neutron-kuttl-tests
2026-03-14 09:20:04.827508 | controller |       NEUTRON_REPO: https://github.com/openstack-k8s-operators/neutron-operator.git
2026-03-14 09:20:04.827513 | controller |       NFS_HOME: /home/nfs
2026-03-14 09:20:04.827517 | controller |       NMSTATE_NAMESPACE: openshift-nmstate
2026-03-14 09:20:04.827522 | controller |       NMSTATE_OPERATOR_GROUP: openshift-nmstate-tn6k8
2026-03-14 09:20:04.827526 | controller |       NMSTATE_SUBSCRIPTION: kubernetes-nmstate-operator
2026-03-14 09:20:04.827530 | controller |       NNCP_ADDITIONAL_HOST_ROUTES: ''
2026-03-14 09:20:04.827534 | controller |       NNCP_BGP_1_INTERFACE: enp7s0
2026-03-14 09:20:04.827539 | controller |       NNCP_BGP_1_IP_ADDRESS: 100.65.4.2
2026-03-14 09:20:04.827543 | controller |       NNCP_BGP_2_INTERFACE: enp8s0
2026-03-14 09:20:04.827547 | controller |       NNCP_BGP_2_IP_ADDRESS: 100.64.4.2
2026-03-14 09:20:04.827551 | controller |       NNCP_BRIDGE: ospbr
2026-03-14 09:20:04.827556 | controller |       NNCP_CLEANUP_TIMEOUT: 120s
2026-03-14 09:20:04.827560 | controller |       NNCP_CTLPLANE_IPV6_ADDRESS_PREFIX: 'fd00:aaaa::'
2026-03-14 09:20:04.827569 | controller |       NNCP_CTLPLANE_IPV6_ADDRESS_SUFFIX: '10'
2026-03-14 09:20:04.827576 | controller |       NNCP_CTLPLANE_IP_ADDRESS_PREFIX: 192.168.122
2026-03-14 09:20:04.827581 | controller |       NNCP_CTLPLANE_IP_ADDRESS_SUFFIX: '10'
2026-03-14 09:20:04.827585 | controller |       NNCP_DNS_SERVER: 192.168.122.1
2026-03-14 09:20:04.827589 | controller |       NNCP_DNS_SERVER_IPV6: fd00:aaaa::1
2026-03-14 09:20:04.827608 | controller |       NNCP_GATEWAY: 192.168.122.1
2026-03-14 09:20:04.827615 | controller |       NNCP_GATEWAY_IPV6: fd00:aaaa::1
2026-03-14 09:20:04.827619 | controller |       NNCP_INTERFACE: enp6s0
2026-03-14 09:20:04.827624 | controller |       NNCP_NODES: ''
2026-03-14 09:20:04.827628 | controller |       NNCP_TIMEOUT: 240s
2026-03-14 09:20:04.827633 | controller |       NOVA: config/samples/nova_v1beta1_nova_collapsed_cell.yaml
2026-03-14 09:20:04.827637 | controller |       NOVA_BRANCH: main
2026-03-14 09:20:04.827641 | controller |       NOVA_COMMIT_HASH: ''
2026-03-14 09:20:04.827645 | controller |       NOVA_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/nova-operator/config/samples/nova_v1beta1_nova_collapsed_cell.yaml
2026-03-14 09:20:04.827650 | controller |       NOVA_IMG: quay.io/openstack-k8s-operators/nova-operator-index:latest
2026-03-14 09:20:04.827654 | controller |       NOVA_REPO: https://github.com/openstack-k8s-operators/nova-operator.git
2026-03-14 09:20:04.827658 | controller |       NUMBER_OF_INSTANCES: '1'
2026-03-14 09:20:04.827663 | controller |       OCP_NETWORK_NAME: crc
2026-03-14 09:20:04.827667 | controller |       OCTAVIA: config/samples/octavia_v1beta1_octavia.yaml
2026-03-14 09:20:04.827671 | controller |       OCTAVIA_BRANCH: main
2026-03-14 09:20:04.827676 | controller |       OCTAVIA_COMMIT_HASH: ''
2026-03-14 09:20:04.827680 | controller |       OCTAVIA_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/octavia-operator/config/samples/octavia_v1beta1_octavia.yaml
2026-03-14 09:20:04.827684 | controller |       OCTAVIA_IMG: quay.io/openstack-k8s-operators/octavia-operator-index:latest
2026-03-14 09:20:04.827689 | controller |       OCTAVIA_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/octavia-operator/kuttl-test.yaml
2026-03-14 09:20:04.827693 | controller |       OCTAVIA_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/octavia-operator/test/kuttl/tests
2026-03-14 09:20:04.827697 | controller |       OCTAVIA_KUTTL_NAMESPACE: octavia-kuttl-tests
2026-03-14 09:20:04.827701 | controller |       OCTAVIA_REPO: https://github.com/openstack-k8s-operators/octavia-operator.git
2026-03-14 09:20:04.827706 | controller |       OKD: 'false'
2026-03-14 09:20:04.827710 | controller |       OPENSTACK_BRANCH: main
2026-03-14 09:20:04.827714 | controller |       OPENSTACK_BUNDLE_IMG: quay.io/openstack-k8s-operators/openstack-operator-bundle:latest
2026-03-14 09:20:04.827718 | controller |       OPENSTACK_COMMIT_HASH: ''
2026-03-14 09:20:04.827723 | controller |       OPENSTACK_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/openstack-operator/config/samples/core_v1beta1_openstackcontrolplane_galera_network_isolation.yaml
2026-03-14 09:20:04.827728 | controller |       OPENSTACK_CRDS_DIR: openstack_crds
2026-03-14 09:20:04.827732 | controller |       OPENSTACK_CTLPLANE: config/samples/core_v1beta1_openstackcontrolplane_galera_network_isolation.yaml
2026-03-14 09:20:04.827736 | controller |       OPENSTACK_IMG: quay.io/openstack-k8s-operators/openstack-operator-index:latest
2026-03-14 09:20:04.827740 | controller |       OPENSTACK_K8S_BRANCH: main
2026-03-14 09:20:04.827745 | controller |       OPENSTACK_K8S_TAG: latest
2026-03-14 09:20:04.827749 | controller |       OPENSTACK_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/openstack-operator/kuttl-test.yaml
2026-03-14 09:20:04.827753 | controller |       OPENSTACK_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/openstack-operator/test/kuttl/tests
2026-03-14 09:20:04.827757 | controller |       OPENSTACK_KUTTL_NAMESPACE: openstack-kuttl-tests
2026-03-14 09:20:04.827762 | controller |       OPENSTACK_NEUTRON_CUSTOM_CONF: ''
2026-03-14 09:20:04.827766 | controller |       OPENSTACK_REPO: https://github.com/openstack-k8s-operators/openstack-operator.git
2026-03-14 09:20:04.827771 | controller |       OPENSTACK_STORAGE_BUNDLE_IMG: quay.io/openstack-k8s-operators/openstack-operator-storage-bundle:latest
2026-03-14 09:20:04.827776 | controller |       OPERATOR_BASE_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator
2026-03-14 09:20:04.827782 | controller |       OPERATOR_CHANNEL: ''
2026-03-14 09:20:04.827787 | controller |       OPERATOR_NAMESPACE: openstack-operators
2026-03-14 09:20:04.827791 | controller |       OPERATOR_SOURCE: ''
2026-03-14 09:20:04.827796 | controller |       OPERATOR_SOURCE_NAMESPACE: ''
2026-03-14 09:20:04.827802 | controller |       OUT: /home/zuul/ci-framework-data/artifacts/manifests
2026-03-14 09:20:04.827806 | controller |       OUTPUT_DIR: /home/zuul/ci-framework-data/artifacts/edpm
2026-03-14 09:20:04.827810 | controller |       OVNCONTROLLER: config/samples/ovn_v1beta1_ovncontroller.yaml
2026-03-14 09:20:04.827815 | controller |       OVNCONTROLLER_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/config/samples/ovn_v1beta1_ovncontroller.yaml
2026-03-14 09:20:04.827819 | controller |       OVNCONTROLLER_NMAP: 'true'
2026-03-14 09:20:04.827824 | controller |       OVNDBS: config/samples/ovn_v1beta1_ovndbcluster.yaml
2026-03-14 09:20:04.827830 | controller |       OVNDBS_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/config/samples/ovn_v1beta1_ovndbcluster.yaml
2026-03-14 09:20:04.827834 | controller |       OVNNORTHD: config/samples/ovn_v1beta1_ovnnorthd.yaml
2026-03-14 09:20:04.827838 | controller |       OVNNORTHD_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/config/samples/ovn_v1beta1_ovnnorthd.yaml
2026-03-14 09:20:04.827843 | controller |       OVN_BRANCH: main
2026-03-14 09:20:04.827847 | controller |       OVN_COMMIT_HASH: ''
2026-03-14 09:20:04.827851 | controller |       OVN_IMG: quay.io/openstack-k8s-operators/ovn-operator-index:latest
2026-03-14 09:20:04.827856 | controller |       OVN_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/kuttl-test.yaml
2026-03-14 09:20:04.827862 | controller |       OVN_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/test/kuttl/tests
2026-03-14 09:20:04.827866 | controller |       OVN_KUTTL_NAMESPACE: ovn-kuttl-tests
2026-03-14 09:20:04.827870 | controller |       OVN_REPO: https://github.com/openstack-k8s-operators/ovn-operator.git
2026-03-14 09:20:04.827875 | controller |       PASSWORD: '12345678'
2026-03-14 09:20:04.827882 | controller |       PLACEMENTAPI: config/samples/placement_v1beta1_placementapi.yaml
2026-03-14 09:20:04.833253 | controller |       PLACEMENTAPI_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/placement-operator/config/samples/placement_v1beta1_placementapi.yaml
2026-03-14 09:20:04.833270 | controller |       PLACEMENTAPI_DEPL_IMG: unused
2026-03-14 09:20:04.833276 | controller |       PLACEMENT_BRANCH: main
2026-03-14 09:20:04.833280 | controller |       PLACEMENT_COMMIT_HASH: ''
2026-03-14 09:20:04.833284 | controller |       PLACEMENT_IMG: quay.io/openstack-k8s-operators/placement-operator-index:latest
2026-03-14 09:20:04.833289 | controller |       PLACEMENT_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/placement-operator/kuttl-test.yaml
2026-03-14 09:20:04.833293 | controller |       PLACEMENT_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/placement-operator/test/kuttl/tests
2026-03-14 09:20:04.833297 | controller |       PLACEMENT_KUTTL_NAMESPACE: placement-kuttl-tests
2026-03-14 09:20:04.833302 | controller |       PLACEMENT_REPO: https://github.com/openstack-k8s-operators/placement-operator.git
2026-03-14 09:20:04.833306 | controller |       PULL_SECRET: /home/zuul/pull-secret.txt
2026-03-14 09:20:04.833310 | controller |       RABBITMQ: docs/examples/default-security-context/rabbitmq.yaml
2026-03-14 09:20:04.833314 | controller |       RABBITMQ_BRANCH: patches
2026-03-14 09:20:04.833319 | controller |       RABBITMQ_COMMIT_HASH: ''
2026-03-14 09:20:04.833323 | controller |       RABBITMQ_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/rabbitmq-operator/docs/examples/default-security-context/rabbitmq.yaml
2026-03-14 09:20:04.833327 | controller |       RABBITMQ_DEPL_IMG: unused
2026-03-14 09:20:04.833331 | controller |       RABBITMQ_IMG: quay.io/openstack-k8s-operators/rabbitmq-cluster-operator-index:latest
2026-03-14 09:20:04.833336 | controller |       RABBITMQ_REPO: https://github.com/openstack-k8s-operators/rabbitmq-cluster-operator.git
2026-03-14 09:20:04.833340 | controller |       REDHAT_OPERATORS: 'false'
2026-03-14 09:20:04.833344 | controller |       REDIS: config/samples/redis_v1beta1_redis.yaml
2026-03-14 09:20:04.833349 | controller |       REDIS_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator-redis/config/samples/redis_v1beta1_redis.yaml
2026-03-14 09:20:04.833356 | controller |       REDIS_DEPL_IMG: unused
2026-03-14 09:20:04.833360 | controller |       RH_REGISTRY_PWD: ''
2026-03-14 09:20:04.833365 | controller |       RH_REGISTRY_USER: ''
2026-03-14 09:20:04.833369 | controller |       SECRET: osp-secret
2026-03-14 09:20:04.833373 | controller |       SG_CORE_DEPL_IMG: unused
2026-03-14 09:20:04.833377 | controller |       STANDALONE_COMPUTE_DRIVER: libvirt
2026-03-14 09:20:04.833381 | controller |       STANDALONE_EXTERNAL_NET_PREFFIX: 172.21.0
2026-03-14 09:20:04.833386 | controller |       STANDALONE_INTERNALAPI_NET_PREFIX: 172.17.0
2026-03-14 09:20:04.833390 | controller |       STANDALONE_STORAGEMGMT_NET_PREFIX: 172.20.0
2026-03-14 09:20:04.833394 | controller |       STANDALONE_STORAGE_NET_PREFIX: 172.18.0
2026-03-14 09:20:04.833398 | controller |       STANDALONE_TENANT_NET_PREFIX: 172.19.0
2026-03-14 09:20:04.833403 | controller |       STORAGEMGMT_HOST_ROUTES: ''
2026-03-14 09:20:04.833407 | controller |       STORAGE_CLASS: local-storage
2026-03-14 09:20:04.833411 | controller |       STORAGE_HOST_ROUTES: ''
2026-03-14 09:20:04.833415 | controller |       SWIFT: config/samples/swift_v1beta1_swift.yaml
2026-03-14 09:20:04.833419 | controller |       SWIFT_BRANCH: main
2026-03-14 09:20:04.833424 | controller |       SWIFT_COMMIT_HASH: ''
2026-03-14 09:20:04.833428 | controller |       SWIFT_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/swift-operator/config/samples/swift_v1beta1_swift.yaml
2026-03-14 09:20:04.833432 | controller |       SWIFT_IMG: quay.io/openstack-k8s-operators/swift-operator-index:latest
2026-03-14 09:20:04.833436 | controller |       SWIFT_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/swift-operator/kuttl-test.yaml
2026-03-14 09:20:04.833441 | controller |       SWIFT_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/swift-operator/test/kuttl/tests
2026-03-14 09:20:04.833445 | controller |       SWIFT_KUTTL_NAMESPACE: swift-kuttl-tests
2026-03-14 09:20:04.833449 | controller |       SWIFT_REPO: https://github.com/openstack-k8s-operators/swift-operator.git
2026-03-14 09:20:04.833453 | controller |       TELEMETRY: config/samples/telemetry_v1beta1_telemetry.yaml
2026-03-14 09:20:04.833457 | controller |       TELEMETRY_BRANCH: main
2026-03-14 09:20:04.833462 | controller |       TELEMETRY_COMMIT_HASH: ''
2026-03-14 09:20:04.833466 | controller |       TELEMETRY_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/telemetry-operator/config/samples/telemetry_v1beta1_telemetry.yaml
2026-03-14 09:20:04.833470 | controller |       TELEMETRY_IMG: quay.io/openstack-k8s-operators/telemetry-operator-index:latest
2026-03-14 09:20:04.833474 | controller |       TELEMETRY_KUTTL_BASEDIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/telemetry-operator
2026-03-14 09:20:04.833478 | controller |       TELEMETRY_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/telemetry-operator/kuttl-test.yaml
2026-03-14 09:20:04.833483 | controller |       TELEMETRY_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/telemetry-operator/test/kuttl/suites
2026-03-14 09:20:04.833487 | controller |       TELEMETRY_KUTTL_NAMESPACE: telemetry-kuttl-tests
2026-03-14 09:20:04.833491 | controller |       TELEMETRY_KUTTL_RELPATH: test/kuttl/suites
2026-03-14 09:20:04.833495 | controller |       TELEMETRY_REPO: https://github.com/openstack-k8s-operators/telemetry-operator.git
2026-03-14 09:20:04.833500 | controller |       TENANT_HOST_ROUTES: ''
2026-03-14 09:20:04.833504 | controller |       TIMEOUT: 300s
2026-03-14 09:20:04.833508 | controller |       TLS_ENABLED: 'false'
2026-03-14 09:20:04.833512 | controller |       WATCHER_BRANCH: ''
2026-03-14 09:20:04.833517 | controller |       WATCHER_REPO: /home/zuul/src/github.com/openstack-k8s-operators/watcher-operator
2026-03-14 09:20:04.833521 | controller |       tripleo_deploy: 'export REGISTRY_PWD:'
2026-03-14 09:20:04.833528 | controller |
2026-03-14 09:20:05.144053 | controller | TASK [install_yamls : Generate make targets install_yamls_path={{ cifmw_install_yamls_repo }}, output_directory={{ cifmw_install_yamls_tasks_out }}] ***
2026-03-14 09:20:05.144101 | controller | Saturday 14 March 2026  09:20:04 +0000 (0:00:00.031)       0:00:56.563 ********
2026-03-14 09:20:05.144107 | controller | Saturday 14 March 2026  09:20:04 +0000 (0:00:00.031)       0:00:56.561 ********
2026-03-14 09:20:05.144119 | controller | ok: [controller]
2026-03-14 09:20:05.161019 | controller |
2026-03-14 09:20:05.161074 | controller | TASK [install_yamls : Debug generate_make module var=cifmw_generate_makes] *****
2026-03-14 09:20:05.161085 | controller | Saturday 14 March 2026  09:20:05 +0000 (0:00:00.310)       0:00:56.873 ********
2026-03-14 09:20:05.161090 | controller | Saturday 14 March 2026  09:20:05 +0000 (0:00:00.310)       0:00:56.872 ********
2026-03-14 09:20:05.161101 | controller | ok: [controller] =>
2026-03-14 09:20:05.161108 | controller |     cifmw_generate_makes:
2026-03-14 09:20:05.161113 | controller |       changed: false
2026-03-14 09:20:05.161118 | controller |       debug:
2026-03-14 09:20:05.161122 | controller |         /home/zuul/src/github.com/openstack-k8s-operators/install_yamls/Makefile:
2026-03-14 09:20:05.161126 | controller |         - all
2026-03-14 09:20:05.161131 | controller |         - help
2026-03-14 09:20:05.161135 | controller |         - cleanup
2026-03-14 09:20:05.161139 | controller |         - deploy_cleanup
2026-03-14 09:20:05.161144 | controller |         - wait
2026-03-14 09:20:05.161148 | controller |         - crc_storage
2026-03-14 09:20:05.161152 | controller |         - crc_storage_cleanup
2026-03-14 09:20:05.161157 | controller |         - crc_storage_release
2026-03-14 09:20:05.161161 | controller |         - crc_storage_with_retries
2026-03-14 09:20:05.161166 | controller |         - crc_storage_cleanup_with_retries
2026-03-14 09:20:05.161170 | controller |         - operator_namespace
2026-03-14 09:20:05.161174 | controller |         - namespace
2026-03-14 09:20:05.161178 | controller |         - namespace_cleanup
2026-03-14 09:20:05.161183 | controller |         - input
2026-03-14 09:20:05.161187 | controller |         - input_cleanup
2026-03-14 09:20:05.161191 | controller |         - crc_bmo_setup
2026-03-14 09:20:05.161195 | controller |         - crc_bmo_cleanup
2026-03-14 09:20:05.161200 | controller |         - mirror_registry
2026-03-14 09:20:05.161204 | controller |         - mirror_registry_secure
2026-03-14 09:20:05.161208 | controller |         - mirror_registry_ca
2026-03-14 09:20:05.161212 | controller |         - mirror_registry_cleanup
2026-03-14 09:20:05.161217 | controller |         - openstack_prep
2026-03-14 09:20:05.161221 | controller |         - openstack
2026-03-14 09:20:05.161225 | controller |         - openstack_wait
2026-03-14 09:20:05.161229 | controller |         - openstack_init
2026-03-14 09:20:05.161233 | controller |         - openstack_cleanup
2026-03-14 09:20:05.161238 | controller |         - openstack_repo
2026-03-14 09:20:05.161242 | controller |         - openstack_deploy_prep
2026-03-14 09:20:05.161246 | controller |         - openstack_deploy
2026-03-14 09:20:05.161250 | controller |         - openstack_wait_deploy
2026-03-14 09:20:05.161254 | controller |         - openstack_deploy_cleanup
2026-03-14 09:20:05.161259 | controller |         - openstack_update_run
2026-03-14 09:20:05.161263 | controller |         - update_services
2026-03-14 09:20:05.161268 | controller |         - update_system
2026-03-14 09:20:05.161272 | controller |         - openstack_patch_version
2026-03-14 09:20:05.161276 | controller |         - edpm_deploy_generate_keys
2026-03-14 09:20:05.161281 | controller |         - edpm_patch_ansible_runner_image
2026-03-14 09:20:05.161285 | controller |         - edpm_deploy_prep
2026-03-14 09:20:05.161289 | controller |         - edpm_deploy_cleanup
2026-03-14 09:20:05.161293 | controller |         - edpm_deploy
2026-03-14 09:20:05.161297 | controller |         - edpm_deploy_baremetal_prep
2026-03-14 09:20:05.161302 | controller |         - edpm_deploy_baremetal
2026-03-14 09:20:05.161306 | controller |         - edpm_wait_deploy_baremetal
2026-03-14 09:20:05.161310 | controller |         - edpm_wait_deploy
2026-03-14 09:20:05.161314 | controller |         - edpm_register_dns
2026-03-14 09:20:05.161319 | controller |         - edpm_nova_discover_hosts
2026-03-14 09:20:05.161323 | controller |         - openstack_crds
2026-03-14 09:20:05.161327 | controller |         - openstack_crds_cleanup
2026-03-14 09:20:05.161331 | controller |         - edpm_deploy_networker_prep
2026-03-14 09:20:05.161335 | controller |         - edpm_deploy_networker_cleanup
2026-03-14 09:20:05.161340 | controller |         - edpm_deploy_networker
2026-03-14 09:20:05.161344 | controller |         - infra_prep
2026-03-14 09:20:05.161348 | controller |         - infra
2026-03-14 09:20:05.161352 | controller |         - infra_cleanup
2026-03-14 09:20:05.161356 | controller |         - dns_deploy_prep
2026-03-14 09:20:05.161360 | controller |         - dns_deploy
2026-03-14 09:20:05.161365 | controller |         - dns_deploy_cleanup
2026-03-14 09:20:05.161369 | controller |         - netconfig_deploy_prep
2026-03-14 09:20:05.161373 | controller |         - netconfig_deploy
2026-03-14 09:20:05.161377 | controller |         - netconfig_deploy_cleanup
2026-03-14 09:20:05.161381 | controller |         - memcached_deploy_prep
2026-03-14 09:20:05.161386 | controller |         - memcached_deploy
2026-03-14 09:20:05.161391 | controller |         - memcached_deploy_cleanup
2026-03-14 09:20:05.161395 | controller |         - keystone_prep
2026-03-14 09:20:05.161400 | controller |         - keystone
2026-03-14 09:20:05.161404 | controller |         - keystone_cleanup
2026-03-14 09:20:05.161408 | controller |         - keystone_deploy_prep
2026-03-14 09:20:05.161412 | controller |         - keystone_deploy
2026-03-14 09:20:05.161416 | controller |         - keystone_deploy_cleanup
2026-03-14 09:20:05.161421 | controller |         - barbican_prep
2026-03-14 09:20:05.161425 | controller |         - barbican
2026-03-14 09:20:05.161429 | controller |         - barbican_cleanup
2026-03-14 09:20:05.161433 | controller |         - barbican_deploy_prep
2026-03-14 09:20:05.161437 | controller |         - barbican_deploy
2026-03-14 09:20:05.161442 | controller |         - barbican_deploy_validate
2026-03-14 09:20:05.161446 | controller |         - barbican_deploy_cleanup
2026-03-14 09:20:05.161450 | controller |         - mariadb
2026-03-14 09:20:05.161454 | controller |         - mariadb_cleanup
2026-03-14 09:20:05.161458 | controller |         - mariadb_deploy_prep
2026-03-14 09:20:05.161464 | controller |         - mariadb_deploy
2026-03-14 09:20:05.161469 | controller |         - mariadb_deploy_cleanup
2026-03-14 09:20:05.161475 | controller |         - placement_prep
2026-03-14 09:20:05.161486 | controller |         - placement
2026-03-14 09:20:05.161496 | controller |         - placement_cleanup
2026-03-14 09:20:05.161502 | controller |         - placement_deploy_prep
2026-03-14 09:20:05.161508 | controller |         - placement_deploy
2026-03-14 09:20:05.161513 | controller |         - placement_deploy_cleanup
2026-03-14 09:20:05.161519 | controller |         - glance_prep
2026-03-14 09:20:05.161524 | controller |         - glance
2026-03-14 09:20:05.161531 | controller |         - glance_cleanup
2026-03-14 09:20:05.161536 | controller |         - glance_deploy_prep
2026-03-14 09:20:05.161545 | controller |         - glance_deploy
2026-03-14 09:20:05.161551 | controller |         - glance_deploy_cleanup
2026-03-14 09:20:05.161557 | controller |         - ovn_prep
2026-03-14 09:20:05.161562 | controller |         - ovn
2026-03-14 09:20:05.161568 | controller |         - ovn_cleanup
2026-03-14 09:20:05.161574 | controller |         - ovn_deploy_prep
2026-03-14 09:20:05.161579 | controller |         - ovn_deploy
2026-03-14 09:20:05.161584 | controller |         - ovn_deploy_cleanup
2026-03-14 09:20:05.161588 | controller |         - neutron_prep
2026-03-14 09:20:05.161592 | controller |         - neutron
2026-03-14 09:20:05.161623 | controller |         - neutron_cleanup
2026-03-14 09:20:05.161631 | controller |         - neutron_deploy_prep
2026-03-14 09:20:05.161636 | controller |         - neutron_deploy
2026-03-14 09:20:05.161640 | controller |         - neutron_deploy_cleanup
2026-03-14 09:20:05.161644 | controller |         - cinder_prep
2026-03-14 09:20:05.161648 | controller |         - cinder
2026-03-14 09:20:05.161652 | controller |         - cinder_cleanup
2026-03-14 09:20:05.161657 | controller |         - cinder_deploy_prep
2026-03-14 09:20:05.161661 | controller |         - cinder_deploy
2026-03-14 09:20:05.161665 | controller |         - cinder_deploy_cleanup
2026-03-14 09:20:05.161670 | controller |         - rabbitmq_prep
2026-03-14 09:20:05.161674 | controller |         - rabbitmq
2026-03-14 09:20:05.161678 | controller |         - rabbitmq_cleanup
2026-03-14 09:20:05.161682 | controller |         - rabbitmq_deploy_prep
2026-03-14 09:20:05.161686 | controller |         - rabbitmq_deploy
2026-03-14 09:20:05.161691 | controller |         - rabbitmq_deploy_cleanup
2026-03-14 09:20:05.161695 | controller |         - ironic_prep
2026-03-14 09:20:05.161699 | controller |         - ironic
2026-03-14 09:20:05.161703 | controller |         - ironic_cleanup
2026-03-14 09:20:05.161707 | controller |         - ironic_deploy_prep
2026-03-14 09:20:05.161711 | controller |         - ironic_deploy
2026-03-14 09:20:05.161715 | controller |         - ironic_deploy_cleanup
2026-03-14 09:20:05.161720 | controller |         - octavia_prep
2026-03-14 09:20:05.161724 | controller |         - octavia
2026-03-14 09:20:05.161728 | controller |         - octavia_cleanup
2026-03-14 09:20:05.161732 | controller |         - octavia_deploy_prep
2026-03-14 09:20:05.161736 | controller |         - octavia_deploy
2026-03-14 09:20:05.161740 | controller |         - octavia_deploy_cleanup
2026-03-14 09:20:05.161745 | controller |         - designate_prep
2026-03-14 09:20:05.161749 | controller |         - designate
2026-03-14 09:20:05.161753 | controller |         - designate_cleanup
2026-03-14 09:20:05.161757 | controller |         - designate_deploy_prep
2026-03-14 09:20:05.161765 | controller |         - designate_deploy
2026-03-14 09:20:05.161769 | controller |         - designate_deploy_cleanup
2026-03-14 09:20:05.161773 | controller |         - nova_prep
2026-03-14 09:20:05.161780 | controller |         - nova
2026-03-14 09:20:05.161784 | controller |         - nova_cleanup
2026-03-14 09:20:05.161789 | controller |         - nova_deploy_prep
2026-03-14 09:20:05.161793 | controller |         - nova_deploy
2026-03-14 09:20:05.161797 | controller |         - nova_deploy_cleanup
2026-03-14 09:20:05.161801 | controller |         - mariadb_kuttl_run
2026-03-14 09:20:05.161805 | controller |         - mariadb_kuttl
2026-03-14 09:20:05.161810 | controller |         - kuttl_db_prep
2026-03-14 09:20:05.161814 | controller |         - kuttl_db_cleanup
2026-03-14 09:20:05.161818 | controller |         - kuttl_common_prep
2026-03-14 09:20:05.161827 | controller |         - kuttl_common_cleanup
2026-03-14 09:20:05.161832 | controller |         - keystone_kuttl_run
2026-03-14 09:20:05.161836 | controller |         - keystone_kuttl
2026-03-14 09:20:05.161840 | controller |         - barbican_kuttl_run
2026-03-14 09:20:05.161844 | controller |         - barbican_kuttl
2026-03-14 09:20:05.161849 | controller |         - placement_kuttl_run
2026-03-14 09:20:05.161853 | controller |         - placement_kuttl
2026-03-14 09:20:05.161857 | controller |         - cinder_kuttl_run
2026-03-14 09:20:05.161861 | controller |         - cinder_kuttl
2026-03-14 09:20:05.161865 | controller |         - neutron_kuttl_run
2026-03-14 09:20:05.161870 | controller |         - neutron_kuttl
2026-03-14 09:20:05.161874 | controller |         - octavia_kuttl_run
2026-03-14 09:20:05.161878 | controller |         - octavia_kuttl
2026-03-14 09:20:05.161884 | controller |         - designate_kuttl
2026-03-14 09:20:05.161889 | controller |         - designate_kuttl_run
2026-03-14 09:20:05.161906 | controller |         - ovn_kuttl_run
2026-03-14 09:20:05.161914 | controller |         - ovn_kuttl
2026-03-14 09:20:05.161921 | controller |         - infra_kuttl_run
2026-03-14 09:20:05.161930 | controller |         - infra_kuttl
2026-03-14 09:20:05.161937 | controller |         - ironic_kuttl_run
2026-03-14 09:20:05.161944 | controller |         - ironic_kuttl
2026-03-14 09:20:05.161951 | controller |         - ironic_kuttl_crc
2026-03-14 09:20:05.161958 | controller |         - heat_kuttl_run
2026-03-14 09:20:05.161966 | controller |         - heat_kuttl
2026-03-14 09:20:05.161973 | controller |         - heat_kuttl_crc
2026-03-14 09:20:05.161981 | controller |         - glance_kuttl_run
2026-03-14 09:20:05.161988 | controller |         - glance_kuttl
2026-03-14 09:20:05.161994 | controller |         - manila_kuttl_run
2026-03-14 09:20:05.161999 | controller |         - manila_kuttl
2026-03-14 09:20:05.162005 | controller |         - swift_kuttl_run
2026-03-14 09:20:05.162010 | controller |         - swift_kuttl
2026-03-14 09:20:05.162016 | controller |         - horizon_kuttl_run
2026-03-14 09:20:05.162021 | controller |         - horizon_kuttl
2026-03-14 09:20:05.162027 | controller |         - openstack_kuttl_prep
2026-03-14 09:20:05.162032 | controller |         - openstack_kuttl_run
2026-03-14 09:20:05.162038 | controller |         - openstack_kuttl_cleanup
2026-03-14 09:20:05.162043 | controller |         - openstack_kuttl
2026-03-14 09:20:05.162049 | controller |         - mariadb_chainsaw_run
2026-03-14 09:20:05.162055 | controller |         - mariadb_chainsaw
2026-03-14 09:20:05.162059 | controller |         - horizon_prep
2026-03-14 09:20:05.162063 | controller |         - horizon
2026-03-14 09:20:05.162067 | controller |         - horizon_cleanup
2026-03-14 09:20:05.162072 | controller |         - horizon_deploy_prep
2026-03-14 09:20:05.162076 | controller |         - horizon_deploy
2026-03-14 09:20:05.162080 | controller |         - horizon_deploy_cleanup
2026-03-14 09:20:05.162084 | controller |         - heat_prep
2026-03-14 09:20:05.162088 | controller |         - heat
2026-03-14 09:20:05.162092 | controller |         - heat_cleanup
2026-03-14 09:20:05.162097 | controller |         - heat_deploy_prep
2026-03-14 09:20:05.162101 | controller |         - heat_deploy
2026-03-14 09:20:05.162105 | controller |         - heat_deploy_cleanup
2026-03-14 09:20:05.162109 | controller |         - baremetal_prep
2026-03-14 09:20:05.162113 | controller |         - baremetal
2026-03-14 09:20:05.162117 | controller |         - baremetal_cleanup
2026-03-14 09:20:05.162122 | controller |         - ceph_help
2026-03-14 09:20:05.162126 | controller |         - ceph
2026-03-14 09:20:05.162130 | controller |         - ceph_cleanup
2026-03-14 09:20:05.162134 | controller |         - rook_prep
2026-03-14 09:20:05.162138 | controller |         - rook
2026-03-14 09:20:05.162142 | controller |         - rook_deploy_prep
2026-03-14 09:20:05.162147 | controller |         - rook_deploy
2026-03-14 09:20:05.162151 | controller |         - rook_crc_disk
2026-03-14 09:20:05.162155 | controller |         - rook_cleanup
2026-03-14 09:20:05.162159 | controller |         - lvms
2026-03-14 09:20:05.162163 | controller |         - nmstate
2026-03-14 09:20:05.162167 | controller |         - nncp
2026-03-14 09:20:05.162172 | controller |         - nncp_cleanup
2026-03-14 09:20:05.162179 | controller |         - netattach
2026-03-14 09:20:05.162184 | controller |         - netattach_cleanup
2026-03-14 09:20:05.162188 | controller |         - metallb
2026-03-14 09:20:05.162192 | controller |         - metallb_config
2026-03-14 09:20:05.162196 | controller |         - metallb_config_cleanup
2026-03-14 09:20:05.162200 | controller |         - metallb_cleanup
2026-03-14 09:20:05.162205 | controller |         - loki
2026-03-14 09:20:05.162209 | controller |         - loki_cleanup
2026-03-14 09:20:05.162213 | controller |         - loki_deploy
2026-03-14 09:20:05.162217 | controller |         - loki_deploy_cleanup
2026-03-14 09:20:05.162221 | controller |         - netobserv
2026-03-14 09:20:05.162225 | controller |         - netobserv_cleanup
2026-03-14 09:20:05.162230 | controller |         - netobserv_deploy
2026-03-14 09:20:05.162234 | controller |         - netobserv_deploy_cleanup
2026-03-14 09:20:05.162238 | controller |         - manila_prep
2026-03-14 09:20:05.162242 | controller |         - manila
2026-03-14 09:20:05.162246 | controller |         - manila_cleanup
2026-03-14 09:20:05.162250 | controller |         - manila_deploy_prep
2026-03-14 09:20:05.162255 | controller |         - manila_deploy
2026-03-14 09:20:05.162259 | controller |         - manila_deploy_cleanup
2026-03-14 09:20:05.162263 | controller |         - telemetry_prep
2026-03-14 09:20:05.162267 | controller |         - telemetry
2026-03-14 09:20:05.162271 | controller |         - telemetry_cleanup
2026-03-14 09:20:05.162275 | controller |         - telemetry_deploy_prep
2026-03-14 09:20:05.162280 | controller |         - telemetry_deploy
2026-03-14 09:20:05.162284 | controller |         - telemetry_deploy_cleanup
2026-03-14 09:20:05.162288 | controller |         - telemetry_kuttl_run
2026-03-14 09:20:05.162294 | controller |         - telemetry_kuttl
2026-03-14 09:20:05.162298 | controller |         - swift_prep
2026-03-14 09:20:05.162303 | controller |         - swift
2026-03-14 09:20:05.162307 | controller |         - swift_cleanup
2026-03-14 09:20:05.162311 | controller |         - swift_deploy_prep
2026-03-14 09:20:05.162315 | controller |         - swift_deploy
2026-03-14 09:20:05.162319 | controller |         - swift_deploy_cleanup
2026-03-14 09:20:05.162323 | controller |         - certmanager
2026-03-14 09:20:05.162327 | controller |         - certmanager_cleanup
2026-03-14 09:20:05.162331 | controller |         - validate_marketplace
2026-03-14 09:20:05.162336 | controller |         - redis_deploy_prep
2026-03-14 09:20:05.162340 | controller |         - redis_deploy
2026-03-14 09:20:05.162344 | controller |         - redis_deploy_cleanup
2026-03-14 09:20:05.162350 | controller |         - set_slower_etcd_profile
2026-03-14 09:20:05.162359 | controller |         /home/zuul/src/github.com/openstack-k8s-operators/install_yamls/devsetup/Makefile:
2026-03-14 09:20:05.162365 | controller |         - help
2026-03-14 09:20:05.162369 | controller |         - download_tools
2026-03-14 09:20:05.162373 | controller |         - nfs
2026-03-14 09:20:05.162377 | controller |         - nfs_cleanup
2026-03-14 09:20:05.162381 | controller |         - crc
2026-03-14 09:20:05.162386 | controller |         - crc_cleanup
2026-03-14 09:20:05.162390 | controller |         - crc_scrub
2026-03-14 09:20:05.162394 | controller |         - crc_attach_default_interface
2026-03-14 09:20:05.162398 | controller |         - crc_attach_default_interface_cleanup
2026-03-14 09:20:05.162403 | controller |         - ipv6_lab_network
2026-03-14 09:20:05.162407 | controller |         - ipv6_lab_network_cleanup
2026-03-14 09:20:05.162411 | controller |         - ipv6_lab_nat64_router
2026-03-14 09:20:05.162415 | controller |         - ipv6_lab_nat64_router_cleanup
2026-03-14 09:20:05.162420 | controller |         - ipv6_lab_sno
2026-03-14 09:20:05.162424 | controller |         - ipv6_lab_sno_cleanup
2026-03-14 09:20:05.162428 | controller |         - ipv6_lab
2026-03-14 09:20:05.162432 | controller |         - ipv6_lab_cleanup
2026-03-14 09:20:05.162436 | controller |         - attach_default_interface
2026-03-14 09:20:05.162440 | controller |         - attach_default_interface_cleanup
2026-03-14 09:20:05.162445 | controller |         - network_isolation_bridge
2026-03-14 09:20:05.162449 | controller |         - network_isolation_bridge_cleanup
2026-03-14 09:20:05.162453 | controller |         - edpm_baremetal_compute
2026-03-14 09:20:05.162457 | controller |         - edpm_compute
2026-03-14 09:20:05.162461 | controller |         - edpm_compute_bootc
2026-03-14 09:20:05.162465 | controller |         - edpm_ansible_runner
2026-03-14 09:20:05.162470 | controller |         - edpm_computes_bgp
2026-03-14 09:20:05.162474 | controller |         - edpm_compute_repos
2026-03-14 09:20:05.162478 | controller |         - edpm_compute_cleanup
2026-03-14 09:20:05.162482 | controller |         - edpm_networker
2026-03-14 09:20:05.162486 | controller |         - edpm_networker_cleanup
2026-03-14 09:20:05.162492 | controller |         - edpm_deploy_instance
2026-03-14 09:20:05.162496 | controller |         - tripleo_deploy
2026-03-14 09:20:05.162500 | controller |         - standalone_deploy
2026-03-14 09:20:05.162505 | controller |         - standalone_sync
2026-03-14 09:20:05.162509 | controller |         - standalone
2026-03-14 09:20:05.162513 | controller |         - standalone_cleanup
2026-03-14 09:20:05.162517 | controller |         - standalone_snapshot
2026-03-14 09:20:05.162521 | controller |         - standalone_revert
2026-03-14 09:20:05.162525 | controller |         - cifmw_prepare
2026-03-14 09:20:05.162530 | controller |         - cifmw_cleanup
2026-03-14 09:20:05.162534 | controller |         - bmaas_network
2026-03-14 09:20:05.162538 | controller |         - bmaas_network_cleanup
2026-03-14 09:20:05.162542 | controller |         - bmaas_route_crc_and_crc_bmaas_networks
2026-03-14 09:20:05.162547 | controller |         - bmaas_route_crc_and_crc_bmaas_networks_cleanup
2026-03-14 09:20:05.162555 | controller |         - bmaas_crc_attach_network
2026-03-14 09:20:05.168626 | controller |         - bmaas_crc_attach_network_cleanup
2026-03-14 09:20:05.168649 | controller |         - bmaas_crc_baremetal_bridge
2026-03-14 09:20:05.168656 | controller |         - bmaas_crc_baremetal_bridge_cleanup
2026-03-14 09:20:05.168662 | controller |         - bmaas_baremetal_net_nad
2026-03-14 09:20:05.168668 | controller |         - bmaas_baremetal_net_nad_cleanup
2026-03-14 09:20:05.168674 | controller |         - bmaas_metallb
2026-03-14 09:20:05.168680 | controller |         - bmaas_metallb_cleanup
2026-03-14 09:20:05.168686 | controller |         - bmaas_virtual_bms
2026-03-14 09:20:05.168692 | controller |         - bmaas_virtual_bms_cleanup
2026-03-14 09:20:05.168698 | controller |         - bmaas_sushy_emulator
2026-03-14 09:20:05.168703 | controller |         - bmaas_sushy_emulator_cleanup
2026-03-14 09:20:05.168710 | controller |         - bmaas_sushy_emulator_wait
2026-03-14 09:20:05.168716 | controller |         - bmaas_generate_nodes_yaml
2026-03-14 09:20:05.168722 | controller |         - bmaas
2026-03-14 09:20:05.168728 | controller |         - bmaas_cleanup
2026-03-14 09:20:05.168734 | controller |       failed: false
2026-03-14 09:20:05.168739 | controller |       success: true
2026-03-14 09:20:05.168750 | controller |
2026-03-14 09:20:05.728276 | controller | TASK [install_yamls : Create the install_yamls parameters file dest={{ cifmw_basedir }}/artifacts/parameters/install-yamls-params.yml, content={{
2026-03-14 09:20:05.728314 | controller |   {
2026-03-14 09:20:05.728322 | controller |   'cifmw_install_yamls_environment': cifmw_install_yamls_environment,
2026-03-14 09:20:05.728329 | controller |   'cifmw_install_yamls_defaults': cifmw_install_yamls_defaults
2026-03-14 09:20:05.728335 | controller |   } | to_nice_yaml
2026-03-14 09:20:05.728341 | controller | }}, mode=0644] ***
2026-03-14 09:20:05.728347 | controller | Saturday 14 March 2026  09:20:05 +0000 (0:00:00.024)       0:00:56.898 ********
2026-03-14 09:20:05.728353 | controller | Saturday 14 March 2026  09:20:05 +0000 (0:00:00.024)       0:00:56.896 ********
2026-03-14 09:20:05.728365 | controller | changed: [controller]
2026-03-14 09:20:05.766139 | controller |
2026-03-14 09:20:05.766178 | controller | TASK [install_yamls : Create empty cifmw_install_yamls_environment if needed cifmw_install_yamls_environment={}] ***
2026-03-14 09:20:05.766188 | controller | Saturday 14 March 2026  09:20:05 +0000 (0:00:00.559)       0:00:57.457 ********
2026-03-14 09:20:05.766195 | controller | Saturday 14 March 2026  09:20:05 +0000 (0:00:00.559)       0:00:57.456 ********
2026-03-14 09:20:05.766207 | controller | skipping: [controller]
2026-03-14 09:20:05.790829 | controller |
2026-03-14 09:20:05.790867 | controller | TASK [discover_latest_image : Set the discovered image vars to empty/null cifmw_discovered_images_dict={}, cifmw_discovered_image_name=None, cifmw_discovered_image_url=None, cifmw_discovered_hash=None, cifmw_discovered_hash_algorithm=None] ***
2026-03-14 09:20:05.790876 | controller | Saturday 14 March 2026  09:20:05 +0000 (0:00:00.037)       0:00:57.495 ********
2026-03-14 09:20:05.790884 | controller | Saturday 14 March 2026  09:20:05 +0000 (0:00:00.037)       0:00:57.494 ********
2026-03-14 09:20:05.790895 | controller | ok: [controller]
2026-03-14 09:20:05.832430 | controller |
2026-03-14 09:20:05.832470 | controller | TASK [discover_latest_image : Discover images _raw_params=per-image.yml] *******
2026-03-14 09:20:05.832478 | controller | Saturday 14 March 2026  09:20:05 +0000 (0:00:00.024)       0:00:57.520 ********
2026-03-14 09:20:05.832485 | controller | Saturday 14 March 2026  09:20:05 +0000 (0:00:00.024)       0:00:57.518 ********
2026-03-14 09:20:05.832504 | controller | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/discover_latest_image/tasks/per-image.yml for controller => (item={'key': 'default', 'value': {'base_url': 'https://cloud.centos.org/centos/9-stream/x86_64/images/', 'qcow_prefix': 'CentOS-Stream-GenericCloud-', 'images_file': 'CHECKSUM'}})
2026-03-14 09:20:05.876973 | controller |
2026-03-14 09:20:05.877010 | controller | TASK [discover_latest_image : Ensure the current image dict entry has the expected data that=['_expected_keys | difference(_given_keys) | length == 0'], fail_msg=The image discovery request dict's {{ _current_image.key }} entry does not match the expected schema. Requests must have the following keys: {{ _expected_keys }}, this entry had: {{ _given_keys }} See the discover_latest_image role's README for more information.] ***
2026-03-14 09:20:05.877023 | controller | Saturday 14 March 2026  09:20:05 +0000 (0:00:00.041)       0:00:57.562 ********
2026-03-14 09:20:05.877030 | controller | Saturday 14 March 2026  09:20:05 +0000 (0:00:00.041)       0:00:57.560 ********
2026-03-14 09:20:05.877042 | controller | ok: [controller] =>
2026-03-14 09:20:06.849254 | controller |     changed: false
2026-03-14 09:20:06.849288 | controller |     msg: All assertions passed
2026-03-14 09:20:06.849296 | controller |
2026-03-14 09:20:06.849301 | controller | TASK [discover_latest_image : Get latest image for the current entry url={{ _current_image.value.base_url }}, image_prefix={{ _current_image.value.qcow_prefix }}, images_file={{ _current_image.value.images_file }}] ***
2026-03-14 09:20:06.849307 | controller | Saturday 14 March 2026  09:20:05 +0000 (0:00:00.044)       0:00:57.606 ********
2026-03-14 09:20:06.849311 | controller | Saturday 14 March 2026  09:20:05 +0000 (0:00:00.044)       0:00:57.605 ********
2026-03-14 09:20:06.849320 | controller | changed: [controller]
2026-03-14 09:20:06.874442 | controller |
2026-03-14 09:20:06.874481 | controller | TASK [discover_latest_image : Append the current return value to the discovered images dict cifmw_discovered_images_dict={{ cifmw_discovered_images_dict | combine({_current_image.key: discovered_image.data}) }}, cacheable=True] ***
2026-03-14 09:20:06.874489 | controller | Saturday 14 March 2026  09:20:06 +0000 (0:00:00.972)       0:00:58.578 ********
2026-03-14 09:20:06.874495 | controller | Saturday 14 March 2026  09:20:06 +0000 (0:00:00.972)       0:00:58.577 ********
2026-03-14 09:20:06.874508 | controller | ok: [controller]
2026-03-14 09:20:06.911383 | controller |
2026-03-14 09:20:06.911420 | controller | TASK [discover_latest_image : Export single-image facts if an image with name 'default' was discovered cifmw_discovered_image_name={{ cifmw_discovered_images_dict.default.image_name }}, cifmw_discovered_image_url={{ cifmw_discovered_images_dict.default.image_url }}, cifmw_discovered_hash={{ cifmw_discovered_images_dict.default.hash }}, cifmw_discovered_hash_algorithm={{ cifmw_discovered_images_dict.default.hash_algorithm }}, cacheable=True] ***
2026-03-14 09:20:06.911427 | controller | Saturday 14 March 2026  09:20:06 +0000 (0:00:00.025)       0:00:58.604 ********
2026-03-14 09:20:06.911432 | controller | Saturday 14 March 2026  09:20:06 +0000 (0:00:00.025)       0:00:58.602 ********
2026-03-14 09:20:06.911442 | controller | ok: [controller]
2026-03-14 09:20:07.476525 | controller |
2026-03-14 09:20:07.476558 | controller | TASK [cifmw_setup : Create artifacts with custom params mode=0644, dest={{ cifmw_basedir }}/artifacts/parameters/custom-params.yml, content={{ ci_framework_params | to_nice_yaml }}] ***
2026-03-14 09:20:07.476565 | controller | Saturday 14 March 2026  09:20:06 +0000 (0:00:00.036)       0:00:58.641 ********
2026-03-14 09:20:07.476570 | controller | Saturday 14 March 2026  09:20:06 +0000 (0:00:00.036)       0:00:58.639 ********
2026-03-14 09:20:07.476580 | controller | changed: [controller]
2026-03-14 09:20:07.506325 | controller |
2026-03-14 09:20:07.506361 | controller | PLAY [Install dev tools] *******************************************************
2026-03-14 09:20:07.506367 | controller |
2026-03-14 09:20:07.506372 | controller | TASK [Assert that operator_name is set that=['operator_name is defined']] ******
2026-03-14 09:20:07.506376 | controller | Saturday 14 March 2026  09:20:07 +0000 (0:00:00.565)       0:00:59.206 ********
2026-03-14 09:20:07.506380 | controller | Saturday 14 March 2026  09:20:07 +0000 (0:00:00.565)       0:00:59.204 ********
2026-03-14 09:20:07.506394 | controller | ok: [controller] =>
2026-03-14 09:20:07.535493 | controller |     changed: false
2026-03-14 09:20:07.535526 | controller |     msg: All assertions passed
2026-03-14 09:20:07.535533 | controller |
2026-03-14 09:20:07.535541 | controller | TASK [Download install_yamls deps name=install_yamls_makes, tasks_from=make_download_tools] ***
2026-03-14 09:20:07.535547 | controller | Saturday 14 March 2026  09:20:07 +0000 (0:00:00.029)       0:00:59.236 ********
2026-03-14 09:20:07.535552 | controller | Saturday 14 March 2026  09:20:07 +0000 (0:00:00.029)       0:00:59.234 ********
2026-03-14 09:20:07.535561 | controller |
2026-03-14 09:20:07.563006 | controller | TASK [install_yamls_makes : Debug make_download_tools_env var=make_download_tools_env] ***
2026-03-14 09:20:07.563042 | controller | Saturday 14 March 2026  09:20:07 +0000 (0:00:00.029)       0:00:59.265 ********
2026-03-14 09:20:07.563048 | controller | Saturday 14 March 2026  09:20:07 +0000 (0:00:00.029)       0:00:59.263 ********
2026-03-14 09:20:07.563057 | controller | skipping: [controller]
2026-03-14 09:20:07.587040 | controller |
2026-03-14 09:20:07.587075 | controller | TASK [install_yamls_makes : Debug make_download_tools_params var=make_download_tools_params] ***
2026-03-14 09:20:07.587081 | controller | Saturday 14 March 2026  09:20:07 +0000 (0:00:00.027)       0:00:59.292 ********
2026-03-14 09:20:07.587086 | controller | Saturday 14 March 2026  09:20:07 +0000 (0:00:00.027)       0:00:59.291 ********
2026-03-14 09:20:07.587096 | controller | skipping: [controller]
2026-03-14 09:20:45.028441 | controller |
2026-03-14 09:20:45.028477 | controller | TASK [install_yamls_makes : Run download_tools output_dir={{ cifmw_basedir }}/artifacts, chdir=/home/zuul/src/github.com/openstack-k8s-operators/install_yamls/devsetup, script=make download_tools, dry_run={{ make_download_tools_dryrun|default(false)|bool }}, extra_args={{ dict((make_download_tools_env|default({})), **(make_download_tools_params|default({}))) }}] ***
2026-03-14 09:20:45.028484 | controller | Saturday 14 March 2026  09:20:07 +0000 (0:00:00.023)       0:00:59.316 ********
2026-03-14 09:20:45.028489 | controller | Saturday 14 March 2026  09:20:07 +0000 (0:00:00.023)       0:00:59.315 ********
2026-03-14 09:20:45.028498 | controller | [WARNING]: conditional statements should not include jinja2 templating
2026-03-14 09:20:45.041949 | controller | delimiters such as {{ }} or {% %}. Found: {{ make_download_tools_until |
2026-03-14 09:20:45.041980 | controller | default(true) }}
2026-03-14 09:20:45.041991 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_000_run_download.log
2026-03-14 09:20:45.121815 | controller | changed: [controller]
2026-03-14 09:20:45.121847 | controller |
2026-03-14 09:20:45.121854 | controller | TASK [run_hook : Assert parameters are valid quiet=True, that=['_list_hooks is not string', '_list_hooks is not mapping', '_list_hooks is iterable', '(hooks | default([])) is not string', '(hooks | default([])) is not mapping', '(hooks | default([])) is iterable']] ***
2026-03-14 09:20:45.121860 | controller | Saturday 14 March 2026  09:20:45 +0000 (0:00:37.455)       0:01:36.771 ********
2026-03-14 09:20:45.121865 | controller | Saturday 14 March 2026  09:20:45 +0000 (0:00:37.455)       0:01:36.770 ********
2026-03-14 09:20:45.121874 | controller | ok: [controller]
2026-03-14 09:20:45.210837 | controller |
2026-03-14 09:20:45.210871 | controller | TASK [run_hook : Assert single hooks are all mappings quiet=True, that=['_not_mapping_hooks | length == 0'], msg=All single hooks must be a list of mappings or a mapping.] ***
2026-03-14 09:20:45.210878 | controller | Saturday 14 March 2026  09:20:45 +0000 (0:00:00.079)       0:01:36.851 ********
2026-03-14 09:20:45.210883 | controller | Saturday 14 March 2026  09:20:45 +0000 (0:00:00.079)       0:01:36.849 ********
2026-03-14 09:20:45.210892 | controller | ok: [controller]
2026-03-14 09:20:45.336248 | controller |
2026-03-14 09:20:45.336287 | controller | TASK [run_hook : Loop on hooks for pre_infra _raw_params={{ hook.type }}.yml] ***
2026-03-14 09:20:45.336296 | controller | Saturday 14 March 2026  09:20:45 +0000 (0:00:00.089)       0:01:36.940 ********
2026-03-14 09:20:45.336304 | controller | Saturday 14 March 2026  09:20:45 +0000 (0:00:00.089)       0:01:36.938 ********
2026-03-14 09:20:45.336317 | controller | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/run_hook/tasks/playbook.yml for controller => (item={'name': 'Download needed tools', 'inventory': 'localhost,', 'connection': 'local', 'type': 'playbook', 'source': '/home/zuul/src/github.com/openstack-k8s-operators/install_yamls/devsetup/download_tools.yaml'})
2026-03-14 09:20:45.381304 | controller |
2026-03-14 09:20:45.381350 | controller | TASK [run_hook : Set playbook path for Download needed tools cifmw_basedir={{ _bdir }}, hook_name={{ _hook_name }}, playbook_path={{ _play | realpath }}, log_path={{ _bdir }}/logs/{{ step }}_{{ _hook_name }}.log, extra_vars=-e namespace={{ cifmw_openstack_namespace }} {%- if hook.extra_vars is defined and hook.extra_vars|length > 0 -%} {% for key,value in hook.extra_vars.items() -%} {%- if key == 'file' %} -e "@{{ value }}" {%- else %} -e "{{ key }}={{ value }}" {%- endif %} {%- endfor %} {%- endif %}] ***
2026-03-14 09:20:45.381364 | controller | Saturday 14 March 2026  09:20:45 +0000 (0:00:00.125)       0:01:37.065 ********
2026-03-14 09:20:45.381374 | controller | Saturday 14 March 2026  09:20:45 +0000 (0:00:00.125)       0:01:37.064 ********
2026-03-14 09:20:45.381390 | controller | ok: [controller]
2026-03-14 09:20:45.592033 | controller |
2026-03-14 09:20:45.592187 | controller | TASK [run_hook : Get file stat path={{ playbook_path }}] ***********************
2026-03-14 09:20:45.592193 | controller | Saturday 14 March 2026  09:20:45 +0000 (0:00:00.045)       0:01:37.110 ********
2026-03-14 09:20:45.592198 | controller | Saturday 14 March 2026  09:20:45 +0000 (0:00:00.045)       0:01:37.109 ********
2026-03-14 09:20:45.592207 | controller | ok: [controller]
2026-03-14 09:20:45.610473 | controller |
2026-03-14 09:20:45.610506 | controller | TASK [run_hook : Fail if playbook doesn't exist msg=Playbook {{ playbook_path }} doesn't seem to exist.] ***
2026-03-14 09:20:45.610513 | controller | Saturday 14 March 2026  09:20:45 +0000 (0:00:00.210)       0:01:37.321 ********
2026-03-14 09:20:45.610517 | controller | Saturday 14 March 2026  09:20:45 +0000 (0:00:00.210)       0:01:37.319 ********
2026-03-14 09:20:45.610527 | controller | skipping: [controller]
2026-03-14 09:20:45.811854 | controller |
2026-03-14 09:20:45.811885 | controller | TASK [run_hook : Get parameters files paths={{
2026-03-14 09:20:45.811892 | controller |   (cifmw_basedir, 'artifacts/parameters') | path_join
2026-03-14 09:20:45.811897 | controller | }}, file_type=file, patterns=*.yml] ***
2026-03-14 09:20:45.811902 | controller | Saturday 14 March 2026  09:20:45 +0000 (0:00:00.018)       0:01:37.340 ********
2026-03-14 09:20:45.811907 | controller | Saturday 14 March 2026  09:20:45 +0000 (0:00:00.018)       0:01:37.338 ********
2026-03-14 09:20:45.811916 | controller | ok: [controller]
2026-03-14 09:20:45.831240 | controller |
2026-03-14 09:20:45.831272 | controller | TASK [run_hook : Add parameters artifacts as extra variables extra_vars={{ extra_vars }} {% for file in cifmw_run_hook_parameters_files.files %} -e "@{{ file.path }}" {%- endfor %}] ***
2026-03-14 09:20:45.831279 | controller | Saturday 14 March 2026  09:20:45 +0000 (0:00:00.201)       0:01:37.541 ********
2026-03-14 09:20:45.831284 | controller | Saturday 14 March 2026  09:20:45 +0000 (0:00:00.201)       0:01:37.540 ********
2026-03-14 09:20:45.831294 | controller | ok: [controller]
2026-03-14 09:20:46.008266 | controller |
2026-03-14 09:20:46.008300 | controller | TASK [run_hook : Ensure log directory exists path={{ log_path | dirname }}, state=directory, mode=0755] ***
2026-03-14 09:20:46.008307 | controller | Saturday 14 March 2026  09:20:45 +0000 (0:00:00.019)       0:01:37.561 ********
2026-03-14 09:20:46.008311 | controller | Saturday 14 March 2026  09:20:45 +0000 (0:00:00.019)       0:01:37.559 ********
2026-03-14 09:20:46.008321 | controller | ok: [controller]
2026-03-14 09:20:46.227431 | controller |
2026-03-14 09:20:46.227466 | controller | TASK [run_hook : Ensure artifacts directory exists path={{ cifmw_basedir }}/artifacts, state=directory, mode=0755] ***
2026-03-14 09:20:46.227472 | controller | Saturday 14 March 2026  09:20:46 +0000 (0:00:00.176)       0:01:37.737 ********
2026-03-14 09:20:46.227477 | controller | Saturday 14 March 2026  09:20:46 +0000 (0:00:00.176)       0:01:37.736 ********
2026-03-14 09:20:46.227486 | controller | ok: [controller]
2026-03-14 09:21:20.381129 | controller |
2026-03-14 09:21:20.381172 | controller | TASK [run_hook : Run hook without retry - Download needed tools] ***************
2026-03-14 09:21:20.381181 | controller | Saturday 14 March 2026  09:20:46 +0000 (0:00:00.219)       0:01:37.957 ********
2026-03-14 09:21:20.381192 | controller | Saturday 14 March 2026  09:20:46 +0000 (0:00:00.219)       0:01:37.955 ********
2026-03-14 09:21:20.381209 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_001_run_hook_without_retry.log
2026-03-14 09:21:20.407462 | controller | changed: [controller]
2026-03-14 09:21:20.407501 | controller |
2026-03-14 09:21:20.407510 | controller | TASK [run_hook : Run hook with retry - Download needed tools] ******************
2026-03-14 09:21:20.407517 | controller | Saturday 14 March 2026  09:21:20 +0000 (0:00:34.153)       0:02:12.110 ********
2026-03-14 09:21:20.407523 | controller | Saturday 14 March 2026  09:21:20 +0000 (0:00:34.153)       0:02:12.109 ********
2026-03-14 09:21:20.407535 | controller | skipping: [controller]
2026-03-14 09:21:20.616045 | controller |
2026-03-14 09:21:20.616077 | controller | TASK [run_hook : Check if we have a file path={{ cifmw_basedir }}/artifacts/{{ step }}_{{ hook_name }}.yml] ***
2026-03-14 09:21:20.616084 | controller | Saturday 14 March 2026  09:21:20 +0000 (0:00:00.026)       0:02:12.137 ********
2026-03-14 09:21:20.616089 | controller | Saturday 14 March 2026  09:21:20 +0000 (0:00:00.026)       0:02:12.135 ********
2026-03-14 09:21:20.616098 | controller | ok: [controller]
2026-03-14 09:21:20.671057 | controller |
2026-03-14 09:21:20.671090 | controller | TASK [run_hook : Load generated content in main playbook file={{ cifmw_basedir }}/artifacts/{{ step }}_{{ hook_name }}.yml] ***
2026-03-14 09:21:20.671096 | controller | Saturday 14 March 2026  09:21:20 +0000 (0:00:00.208)       0:02:12.345 ********
2026-03-14 09:21:20.671101 | controller | Saturday 14 March 2026  09:21:20 +0000 (0:00:00.208)       0:02:12.344 ********
2026-03-14 09:21:20.671111 | controller | skipping: [controller]
2026-03-14 09:21:20.714935 | controller |
2026-03-14 09:21:20.714969 | controller | PLAY [Prepare host virtualization] *********************************************
2026-03-14 09:21:20.714976 | controller |
2026-03-14 09:21:20.714981 | controller | TASK [cifmw_setup : Load parameters files dir={{ cifmw_basedir }}/artifacts/parameters] ***
2026-03-14 09:21:20.714986 | controller | Saturday 14 March 2026  09:21:20 +0000 (0:00:00.055)       0:02:12.401 ********
2026-03-14 09:21:20.714990 | controller | Saturday 14 March 2026  09:21:20 +0000 (0:00:00.055)       0:02:12.399 ********
2026-03-14 09:21:20.714999 | controller | ok: [controller]
2026-03-14 09:21:20.737431 | controller |
2026-03-14 09:21:20.737464 | controller | TASK [Ensure libvirt is present/configured name=libvirt_manager] ***************
2026-03-14 09:21:20.737470 | controller | Saturday 14 March 2026  09:21:20 +0000 (0:00:00.043)       0:02:12.444 ********
2026-03-14 09:21:20.737475 | controller | Saturday 14 March 2026  09:21:20 +0000 (0:00:00.043)       0:02:12.443 ********
2026-03-14 09:21:20.737484 | controller | skipping: [controller]
2026-03-14 09:21:20.812709 | controller |
2026-03-14 09:21:20.812745 | controller | TASK [Perpare OpenShift provisioner node name=openshift_provisioner_node] ******
2026-03-14 09:21:20.812751 | controller | Saturday 14 March 2026  09:21:20 +0000 (0:00:00.022)       0:02:12.467 ********
2026-03-14 09:21:20.812756 | controller | Saturday 14 March 2026  09:21:20 +0000 (0:00:00.022)       0:02:12.465 ********
2026-03-14 09:21:20.812766 | controller | skipping: [controller]
2026-03-14 09:21:20.884335 | controller |
2026-03-14 09:21:20.884375 | controller | PLAY [Build dataset hook] ******************************************************
2026-03-14 09:21:20.884384 | controller |
2026-03-14 09:21:20.884391 | controller | TASK [cifmw_setup : Load parameters files dir={{ cifmw_basedir }}/artifacts/parameters] ***
2026-03-14 09:21:20.884398 | controller | Saturday 14 March 2026  09:21:20 +0000 (0:00:00.075)       0:02:12.542 ********
2026-03-14 09:21:20.884404 | controller | Saturday 14 March 2026  09:21:20 +0000 (0:00:00.075)       0:02:12.540 ********
2026-03-14 09:21:20.884417 | controller | ok: [controller]
2026-03-14 09:21:21.116955 | controller |
2026-03-14 09:21:21.117030 | controller | TASK [networking_mapper : Check for Networking Environment Definition file existence path={{ cifmw_networking_mapper_networking_env_def_path }}] ***
2026-03-14 09:21:21.117039 | controller | Saturday 14 March 2026  09:21:20 +0000 (0:00:00.071)       0:02:12.613 ********
2026-03-14 09:21:21.117045 | controller | Saturday 14 March 2026  09:21:20 +0000 (0:00:00.071)       0:02:12.612 ********
2026-03-14 09:21:21.117064 | controller | ok: [controller]
2026-03-14 09:21:21.145906 | controller |
2026-03-14 09:21:21.145962 | controller | TASK [networking_mapper : Check for Networking Definition file existance that=['_net_env_def_stat.stat.exists'], msg=Ensure that the Networking Environment Definition file exists in {{ cifmw_networking_mapper_networking_env_def_path }}, quiet=True] ***
2026-03-14 09:21:21.145974 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.232)       0:02:12.846 ********
2026-03-14 09:21:21.145980 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.232)       0:02:12.844 ********
2026-03-14 09:21:21.145991 | controller | skipping: [controller]
2026-03-14 09:21:21.175036 | controller |
2026-03-14 09:21:21.175099 | controller | TASK [networking_mapper : Load the Networking Definition from file path={{ cifmw_networking_mapper_networking_env_def_path }}] ***
2026-03-14 09:21:21.175106 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.029)       0:02:12.875 ********
2026-03-14 09:21:21.175111 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.029)       0:02:12.873 ********
2026-03-14 09:21:21.175123 | controller | skipping: [controller]
2026-03-14 09:21:21.207221 | controller |
2026-03-14 09:21:21.207293 | controller | TASK [networking_mapper : Set cifmw_networking_env_definition is present cifmw_networking_env_definition={{
2026-03-14 09:21:21.207299 | controller |   _net_env_def_slurp['content'] |
2026-03-14 09:21:21.207304 | controller |   b64decode |
2026-03-14 09:21:21.207309 | controller |   from_yaml
2026-03-14 09:21:21.207313 | controller | }}, cacheable=True] ***
2026-03-14 09:21:21.207318 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.029)       0:02:12.904 ********
2026-03-14 09:21:21.207322 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.029)       0:02:12.903 ********
2026-03-14 09:21:21.207338 | controller | skipping: [controller]
2026-03-14 09:21:21.228890 | controller |
2026-03-14 09:21:21.228942 | controller | TASK [Deploy OCP using Hive name=hive] *****************************************
2026-03-14 09:21:21.228949 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.032)       0:02:12.936 ********
2026-03-14 09:21:21.228957 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.032)       0:02:12.935 ********
2026-03-14 09:21:21.228970 | controller | skipping: [controller]
2026-03-14 09:21:21.250491 | controller |
2026-03-14 09:21:21.250549 | controller | TASK [Prepare CRC name=rhol_crc] ***********************************************
2026-03-14 09:21:21.250558 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.021)       0:02:12.958 ********
2026-03-14 09:21:21.250565 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.021)       0:02:12.957 ********
2026-03-14 09:21:21.250579 | controller | skipping: [controller]
2026-03-14 09:21:21.276521 | controller |
2026-03-14 09:21:21.276584 | controller | TASK [Deploy OpenShift cluster using dev-scripts name=devscripts] **************
2026-03-14 09:21:21.276592 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.021)       0:02:12.980 ********
2026-03-14 09:21:21.276612 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.021)       0:02:12.978 ********
2026-03-14 09:21:21.276628 | controller | skipping: [controller]
2026-03-14 09:21:21.482995 | controller |
2026-03-14 09:21:21.483028 | controller | TASK [openshift_login : Ensure output directory exists path={{ cifmw_openshift_login_basedir }}/artifacts, state=directory, mode=0755] ***
2026-03-14 09:21:21.483035 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.025)       0:02:13.006 ********
2026-03-14 09:21:21.483040 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.025)       0:02:13.004 ********
2026-03-14 09:21:21.483048 | controller | ok: [controller]
2026-03-14 09:21:21.517908 | controller |
2026-03-14 09:21:21.517942 | controller | TASK [openshift_login : OpenShift login _raw_params=login.yml] *****************
2026-03-14 09:21:21.517948 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.206)       0:02:13.212 ********
2026-03-14 09:21:21.517953 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.206)       0:02:13.211 ********
2026-03-14 09:21:21.517962 | controller | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/openshift_login/tasks/login.yml for controller
2026-03-14 09:21:21.540513 | controller |
2026-03-14 09:21:21.540553 | controller | TASK [openshift_login : Check if the password file is present path={{ cifmw_openshift_login_password_file | default(cifmw_openshift_password_file) }}] ***
2026-03-14 09:21:21.540563 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.034)       0:02:13.247 ********
2026-03-14 09:21:21.540581 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.034)       0:02:13.246 ********
2026-03-14 09:21:21.540601 | controller | skipping: [controller]
2026-03-14 09:21:21.561485 | controller |
2026-03-14 09:21:21.561516 | controller | TASK [openshift_login : Fetch user password content src={{ cifmw_openshift_login_password_file | default(cifmw_openshift_password_file) }}] ***
2026-03-14 09:21:21.561522 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.022)       0:02:13.270 ********
2026-03-14 09:21:21.561527 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.022)       0:02:13.268 ********
2026-03-14 09:21:21.561536 | controller | skipping: [controller]
2026-03-14 09:21:21.590000 | controller |
2026-03-14 09:21:21.590046 | controller | TASK [openshift_login : Set user password as a fact cifmw_openshift_login_password={{ cifmw_openshift_login_password_file_slurp.content | b64decode }}, cacheable=True] ***
2026-03-14 09:21:21.590057 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.021)       0:02:13.291 ********
2026-03-14 09:21:21.590064 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.021)       0:02:13.289 ********
2026-03-14 09:21:21.590077 | controller | skipping: [controller]
2026-03-14 09:21:21.627524 | controller |
2026-03-14 09:21:21.627567 | controller | TASK [openshift_login : Set role variables cifmw_openshift_login_kubeconfig={{
2026-03-14 09:21:21.627578 | controller |  cifmw_openshift_login_kubeconfig |
2026-03-14 09:21:21.627586 | controller |  default(cifmw_openshift_kubeconfig) |
2026-03-14 09:21:21.627595 | controller |  default(
2026-03-14 09:21:21.627610 | controller |    ansible_env.KUBECONFIG if 'KUBECONFIG' in ansible_env else
2026-03-14 09:21:21.627618 | controller |    cifmw_openshift_login_kubeconfig_default_path
2026-03-14 09:21:21.627627 | controller |  ) | trim
2026-03-14 09:21:21.627663 | controller | }}, cifmw_openshift_login_user={{ cifmw_openshift_login_user | default(cifmw_openshift_user) | default(omit) }}, cifmw_openshift_login_password={{ cifmw_openshift_login_password | default(cifmw_openshift_password) | default(omit) }}, cifmw_openshift_login_api={{ cifmw_openshift_login_api | default(cifmw_openshift_api) | default(omit) }}, cifmw_openshift_login_cert_login={{ cifmw_openshift_login_cert_login | default(false)}}, cifmw_openshift_login_provided_token={{ cifmw_openshift_provided_token | default(omit) }}, cacheable=True] ***
2026-03-14 09:21:21.627677 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.027)       0:02:13.319 ********
2026-03-14 09:21:21.627686 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.027)       0:02:13.317 ********
2026-03-14 09:21:21.627701 | controller | ok: [controller]
2026-03-14 09:21:21.803618 | controller |
2026-03-14 09:21:21.803666 | controller | TASK [openshift_login : Check if kubeconfig exists path={{ cifmw_openshift_login_kubeconfig }}] ***
2026-03-14 09:21:21.803673 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.037)       0:02:13.357 ********
2026-03-14 09:21:21.803678 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.037)       0:02:13.355 ********
2026-03-14 09:21:21.803687 | controller | ok: [controller]
2026-03-14 09:21:21.828167 | controller |
2026-03-14 09:21:21.828198 | controller | TASK [openshift_login : Assert that enough data is provided to log in to OpenShift that=cifmw_openshift_login_kubeconfig_stat.stat.exists or (cifmw_openshift_login_provided_token is defined and cifmw_openshift_login_provided_token != '') or (
2026-03-14 09:21:21.828205 | controller |   (cifmw_openshift_login_user is defined) and
2026-03-14 09:21:21.828210 | controller |   (cifmw_openshift_login_password is defined) and
2026-03-14 09:21:21.828215 | controller |   (cifmw_openshift_login_api is defined)
2026-03-14 09:21:21.828220 | controller | ), msg=If an existing kubeconfig is not provided user/pwd or provided/initial token and API URL must be given] ***
2026-03-14 09:21:21.828224 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.176)       0:02:13.533 ********
2026-03-14 09:21:21.828229 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.176)       0:02:13.531 ********
2026-03-14 09:21:21.828237 | controller | ok: [controller] =>
2026-03-14 09:21:21.848027 | controller |     changed: false
2026-03-14 09:21:21.848058 | controller |     msg: All assertions passed
2026-03-14 09:21:21.848064 | controller |
2026-03-14 09:21:21.848069 | controller | TASK [openshift_login : Fetch kubeconfig content src={{ cifmw_openshift_login_kubeconfig }}] ***
2026-03-14 09:21:21.848079 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.024)       0:02:13.558 ********
2026-03-14 09:21:21.848083 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.024)       0:02:13.556 ********
2026-03-14 09:21:21.848097 | controller | skipping: [controller]
2026-03-14 09:21:21.867412 | controller |
2026-03-14 09:21:21.867471 | controller | TASK [openshift_login : Fetch x509 key based users cifmw_openshift_login_key_based_users={{
2026-03-14 09:21:21.867478 | controller |   (
2026-03-14 09:21:21.867482 | controller |     cifmw_openshift_login_kubeconfig_content_b64.content |
2026-03-14 09:21:21.867487 | controller |     b64decode |
2026-03-14 09:21:21.867491 | controller |     from_yaml
2026-03-14 09:21:21.867496 | controller |   ). users | default([]) |
2026-03-14 09:21:21.867503 | controller |   selectattr('user.client-certificate-data', 'defined') |
2026-03-14 09:21:21.867507 | controller |   map(attribute="name") |
2026-03-14 09:21:21.867512 | controller |   map("split", "/") |
2026-03-14 09:21:21.867516 | controller |   map("first")
2026-03-14 09:21:21.867520 | controller | }}, cacheable=True] ***
2026-03-14 09:21:21.867525 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.019)       0:02:13.577 ********
2026-03-14 09:21:21.867529 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.019)       0:02:13.576 ********
2026-03-14 09:21:21.867538 | controller | skipping: [controller]
2026-03-14 09:21:21.886600 | controller |
2026-03-14 09:21:21.886631 | controller | TASK [openshift_login : Assign key based user if not provided and available cifmw_openshift_login_user={{
2026-03-14 09:21:21.886654 | controller |   (cifmw_openshift_login_assume_cert_system_user | ternary('system:', '')) +
2026-03-14 09:21:21.886660 | controller |   (cifmw_openshift_login_key_based_users | map('replace', 'system:', '') | unique | first)
2026-03-14 09:21:21.886665 | controller | }}, cifmw_openshift_login_cert_login=True, cacheable=True] ***
2026-03-14 09:21:21.886669 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.019)       0:02:13.597 ********
2026-03-14 09:21:21.886676 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.019)       0:02:13.595 ********
2026-03-14 09:21:21.886686 | controller | skipping: [controller]
2026-03-14 09:21:21.909517 | controller |
2026-03-14 09:21:21.909550 | controller | TASK [openshift_login : Set the retry count cifmw_openshift_login_retries_cnt={{
2026-03-14 09:21:21.909556 | controller |   0 if cifmw_openshift_login_retries_cnt is undefined else
2026-03-14 09:21:21.909561 | controller |   cifmw_openshift_login_retries_cnt|int + 1
2026-03-14 09:21:21.909566 | controller | }}] ***
2026-03-14 09:21:21.909571 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.019)       0:02:13.616 ********
2026-03-14 09:21:21.909575 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.019)       0:02:13.614 ********
2026-03-14 09:21:21.909584 | controller | ok: [controller]
2026-03-14 09:21:21.934743 | controller |
2026-03-14 09:21:21.934777 | controller | TASK [openshift_login : Fetch token _raw_params=try_login.yml] *****************
2026-03-14 09:21:21.934783 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.022)       0:02:13.639 ********
2026-03-14 09:21:21.934787 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.022)       0:02:13.637 ********
2026-03-14 09:21:21.934796 | controller | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/openshift_login/tasks/try_login.yml for controller
2026-03-14 09:21:21.950996 | controller |
2026-03-14 09:21:21.951038 | controller | TASK [openshift_login : Try get OpenShift access token _raw_params=oc whoami -t] ***
2026-03-14 09:21:21.951045 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.025)       0:02:13.664 ********
2026-03-14 09:21:21.951057 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.025)       0:02:13.662 ********
2026-03-14 09:21:21.951066 | controller | skipping: [controller]
2026-03-14 09:21:22.544186 | controller |
2026-03-14 09:21:22.544257 | controller | TASK [openshift_login : Fetch OpenShift token output_dir={{ cifmw_openshift_login_basedir }}/artifacts, script=oc login {%- if cifmw_openshift_login_provided_token is not defined %} {%- if cifmw_openshift_login_user is defined %} -u {{ cifmw_openshift_login_user }} {%- endif %} {%- if cifmw_openshift_login_password is defined %} -p {{ cifmw_openshift_login_password }} {%- endif %} {% else %} --token={{ cifmw_openshift_login_provided_token }} {%- endif %} {%- if cifmw_openshift_login_skip_tls_verify|bool %} --insecure-skip-tls-verify=true {%- endif %} {%- if cifmw_openshift_login_api is defined %} {{ cifmw_openshift_login_api }} {%- endif %}] ***
2026-03-14 09:21:22.544275 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.016)       0:02:13.680 ********
2026-03-14 09:21:22.544280 | controller | Saturday 14 March 2026  09:21:21 +0000 (0:00:00.016)       0:02:13.679 ********
2026-03-14 09:21:22.544294 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_002_fetch_openshift.log
2026-03-14 09:21:22.572047 | controller | changed: [controller]
2026-03-14 09:21:22.572108 | controller |
2026-03-14 09:21:22.572114 | controller | TASK [openshift_login : Ensure kubeconfig is provided that=cifmw_openshift_login_kubeconfig != ""] ***
2026-03-14 09:21:22.572121 | controller | Saturday 14 March 2026  09:21:22 +0000 (0:00:00.592)       0:02:14.273 ********
2026-03-14 09:21:22.572166 | controller | Saturday 14 March 2026  09:21:22 +0000 (0:00:00.592)       0:02:14.272 ********
2026-03-14 09:21:22.572182 | controller | ok: [controller] =>
2026-03-14 09:21:22.887330 | controller |     changed: false
2026-03-14 09:21:22.887377 | controller |     msg: All assertions passed
2026-03-14 09:21:22.887384 | controller |
2026-03-14 09:21:22.887389 | controller | TASK [openshift_login : Fetch new OpenShift access token _raw_params=oc whoami -t] ***
2026-03-14 09:21:22.887394 | controller | Saturday 14 March 2026  09:21:22 +0000 (0:00:00.027)       0:02:14.301 ********
2026-03-14 09:21:22.887399 | controller | Saturday 14 March 2026  09:21:22 +0000 (0:00:00.027)       0:02:14.299 ********
2026-03-14 09:21:22.887410 | controller | changed: [controller]
2026-03-14 09:21:22.915428 | controller |
2026-03-14 09:21:22.915480 | controller | TASK [openshift_login : Set new OpenShift token cifmw_openshift_login_token={{
2026-03-14 09:21:22.915486 | controller |   (not cifmw_openshift_login_new_token_out.skipped | default(false)) |
2026-03-14 09:21:22.915491 | controller |   ternary(cifmw_openshift_login_new_token_out.stdout, cifmw_openshift_login_whoami_out.stdout)
2026-03-14 09:21:22.915496 | controller | }}, cacheable=True] ***
2026-03-14 09:21:22.915501 | controller | Saturday 14 March 2026  09:21:22 +0000 (0:00:00.315)       0:02:14.617 ********
2026-03-14 09:21:22.915505 | controller | Saturday 14 March 2026  09:21:22 +0000 (0:00:00.315)       0:02:14.615 ********
2026-03-14 09:21:22.915518 | controller | ok: [controller]
2026-03-14 09:21:23.214828 | controller |
2026-03-14 09:21:23.214878 | controller | TASK [openshift_login : Fetch OpenShift API URL _raw_params=oc whoami --show-server=true] ***
2026-03-14 09:21:23.214884 | controller | Saturday 14 March 2026  09:21:22 +0000 (0:00:00.028)       0:02:14.645 ********
2026-03-14 09:21:23.214889 | controller | Saturday 14 March 2026  09:21:22 +0000 (0:00:00.028)       0:02:14.643 ********
2026-03-14 09:21:23.214901 | controller | changed: [controller]
2026-03-14 09:21:23.543804 | controller |
2026-03-14 09:21:23.543835 | controller | TASK [openshift_login : Fetch OpenShift kubeconfig context _raw_params=oc whoami -c] ***
2026-03-14 09:21:23.543841 | controller | Saturday 14 March 2026  09:21:23 +0000 (0:00:00.299)       0:02:14.944 ********
2026-03-14 09:21:23.543846 | controller | Saturday 14 March 2026  09:21:23 +0000 (0:00:00.299)       0:02:14.942 ********
2026-03-14 09:21:23.543855 | controller | changed: [controller]
2026-03-14 09:21:23.868334 | controller |
2026-03-14 09:21:23.868376 | controller | TASK [openshift_login : Fetch OpenShift current user _raw_params=oc whoami] ****
2026-03-14 09:21:23.868384 | controller | Saturday 14 March 2026  09:21:23 +0000 (0:00:00.329)       0:02:15.273 ********
2026-03-14 09:21:23.868391 | controller | Saturday 14 March 2026  09:21:23 +0000 (0:00:00.329)       0:02:15.272 ********
2026-03-14 09:21:23.868403 | controller | changed: [controller]
2026-03-14 09:21:23.908902 | controller |
2026-03-14 09:21:23.908943 | controller | TASK [openshift_login : Set OpenShift user, context and API facts cifmw_openshift_login_api={{ cifmw_openshift_login_api_out.stdout }}, cifmw_openshift_login_context={{ cifmw_openshift_login_context_out.stdout }}, cifmw_openshift_login_user={{ _oauth_user }}, cifmw_openshift_kubeconfig={{ cifmw_openshift_login_kubeconfig }}, cifmw_openshift_api={{ cifmw_openshift_login_api_out.stdout }}, cifmw_openshift_context={{ cifmw_openshift_login_context_out.stdout }}, cifmw_openshift_user={{ _oauth_user }}, cifmw_openshift_token={{ cifmw_openshift_login_token | default(omit) }}, cifmw_install_yamls_environment={{  ( cifmw_install_yamls_environment |
2026-03-14 09:21:23.909137 | controller |       combine({'KUBECONFIG': cifmw_openshift_login_kubeconfig})
2026-03-14 09:21:23.909144 | controller |     ) if cifmw_install_yamls_environment is defined else omit
2026-03-14 09:21:23.909150 | controller | }}, cacheable=True] ***
2026-03-14 09:21:23.909157 | controller | Saturday 14 March 2026  09:21:23 +0000 (0:00:00.324)       0:02:15.597 ********
2026-03-14 09:21:23.909163 | controller | Saturday 14 March 2026  09:21:23 +0000 (0:00:00.324)       0:02:15.596 ********
2026-03-14 09:21:23.909179 | controller | ok: [controller]
2026-03-14 09:21:24.471303 | controller |
2026-03-14 09:21:24.471343 | controller | TASK [openshift_login : Create the openshift_login parameters file dest={{ cifmw_basedir }}/artifacts/parameters/openshift-login-params.yml, content={{ cifmw_openshift_login_params_content | from_yaml | to_nice_yaml }}, mode=0600] ***
2026-03-14 09:21:24.471355 | controller | Saturday 14 March 2026  09:21:23 +0000 (0:00:00.040)       0:02:15.638 ********
2026-03-14 09:21:24.471363 | controller | Saturday 14 March 2026  09:21:23 +0000 (0:00:00.040)       0:02:15.636 ********
2026-03-14 09:21:24.471376 | controller | changed: [controller]
2026-03-14 09:21:24.804914 | controller |
2026-03-14 09:21:24.804946 | controller | TASK [openshift_login : Read the install yamls parameters file path={{ cifmw_basedir }}/artifacts/parameters/install-yamls-params.yml] ***
2026-03-14 09:21:24.804953 | controller | Saturday 14 March 2026  09:21:24 +0000 (0:00:00.562)       0:02:16.200 ********
2026-03-14 09:21:24.804958 | controller | Saturday 14 March 2026  09:21:24 +0000 (0:00:00.562)       0:02:16.199 ********
2026-03-14 09:21:24.804967 | controller | ok: [controller]
2026-03-14 09:21:25.369532 | controller |
2026-03-14 09:21:25.369564 | controller | TASK [openshift_login : Append the KUBECONFIG to the install yamls parameters content={{
2026-03-14 09:21:25.369570 | controller |   cifmw_openshift_login_install_yamls_artifacts_slurp['content'] | b64decode | from_yaml |
2026-03-14 09:21:25.369574 | controller |   combine(
2026-03-14 09:21:25.369579 | controller |     {
2026-03-14 09:21:25.369583 | controller |       'cifmw_install_yamls_environment': {
2026-03-14 09:21:25.369588 | controller |         'KUBECONFIG': cifmw_openshift_login_kubeconfig
2026-03-14 09:21:25.369593 | controller |       }
2026-03-14 09:21:25.369601 | controller |     }, recursive=true) | to_nice_yaml
2026-03-14 09:21:25.369605 | controller | }}, dest={{ cifmw_basedir }}/artifacts/parameters/install-yamls-params.yml, mode=0600] ***
2026-03-14 09:21:25.369610 | controller | Saturday 14 March 2026  09:21:24 +0000 (0:00:00.333)       0:02:16.534 ********
2026-03-14 09:21:25.369615 | controller | Saturday 14 March 2026  09:21:24 +0000 (0:00:00.333)       0:02:16.533 ********
2026-03-14 09:21:25.369643 | controller | changed: [controller]
2026-03-14 09:21:25.610048 | controller |
2026-03-14 09:21:25.610108 | controller | TASK [openshift_setup : Ensure output directory exists path={{ cifmw_openshift_setup_basedir }}/artifacts, state=directory, mode=0755] ***
2026-03-14 09:21:25.610122 | controller | Saturday 14 March 2026  09:21:25 +0000 (0:00:00.564)       0:02:17.099 ********
2026-03-14 09:21:25.610132 | controller | Saturday 14 March 2026  09:21:25 +0000 (0:00:00.564)       0:02:17.097 ********
2026-03-14 09:21:25.610150 | controller | ok: [controller]
2026-03-14 09:21:25.654264 | controller |
2026-03-14 09:21:25.654313 | controller | TASK [openshift_setup : Fetch namespaces to create cifmw_openshift_setup_namespaces={{
2026-03-14 09:21:25.654326 | controller |   ((
2026-03-14 09:21:25.654336 | controller |       ([cifmw_install_yamls_defaults['NAMESPACE']] +
2026-03-14 09:21:25.654347 | controller |         ([cifmw_install_yamls_defaults['OPERATOR_NAMESPACE']] if 'OPERATOR_NAMESPACE' is in cifmw_install_yamls_defaults else [])
2026-03-14 09:21:25.654357 | controller |       ) if cifmw_install_yamls_defaults is defined else []
2026-03-14 09:21:25.654366 | controller |   ) + cifmw_openshift_setup_create_namespaces) | unique
2026-03-14 09:21:25.654376 | controller | }}] ***
2026-03-14 09:21:25.654386 | controller | Saturday 14 March 2026  09:21:25 +0000 (0:00:00.239)       0:02:17.339 ********
2026-03-14 09:21:25.654396 | controller | Saturday 14 March 2026  09:21:25 +0000 (0:00:00.239)       0:02:17.337 ********
2026-03-14 09:21:25.654413 | controller | ok: [controller]
2026-03-14 09:21:27.855440 | controller |
2026-03-14 09:21:27.855478 | controller | TASK [openshift_setup : Create required namespaces kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit) }}, name={{ item }}, kind=Namespace, state=present] ***
2026-03-14 09:21:27.855491 | controller | Saturday 14 March 2026  09:21:25 +0000 (0:00:00.044)       0:02:17.383 ********
2026-03-14 09:21:27.855498 | controller | Saturday 14 March 2026  09:21:25 +0000 (0:00:00.044)       0:02:17.381 ********
2026-03-14 09:21:27.855510 | controller | changed: [controller] => (item=openstack)
2026-03-14 09:21:27.877519 | controller | changed: [controller] => (item=openstack-operators)
2026-03-14 09:21:27.877553 | controller |
2026-03-14 09:21:27.877562 | controller | TASK [openshift_setup : Get internal OpenShift registry route kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, kind=Route, name=default-route, namespace=openshift-image-registry] ***
2026-03-14 09:21:27.877569 | controller | Saturday 14 March 2026  09:21:27 +0000 (0:00:02.201)       0:02:19.585 ********
2026-03-14 09:21:27.877575 | controller | Saturday 14 March 2026  09:21:27 +0000 (0:00:02.201)       0:02:19.583 ********
2026-03-14 09:21:27.877587 | controller | skipping: [controller]
2026-03-14 09:21:27.906985 | controller |
2026-03-14 09:21:27.907053 | controller | TASK [openshift_setup : Allow anonymous image-pulls in CRC registry for targeted namespaces state=present, kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, definition={'kind': 'RoleBinding', 'apiVersion': 'rbac.authorization.k8s.io/v1', 'metadata': {'name': 'system:image-puller', 'namespace': '{{ item }}'}, 'subjects': [{'kind': 'User', 'name': 'system:anonymous'}, {'kind': 'User', 'name': 'system:unauthenticated'}], 'roleRef': {'kind': 'ClusterRole', 'name': 'system:image-puller'}}] ***
2026-03-14 09:21:27.907064 | controller | Saturday 14 March 2026  09:21:27 +0000 (0:00:00.022)       0:02:19.607 ********
2026-03-14 09:21:27.907072 | controller | Saturday 14 March 2026  09:21:27 +0000 (0:00:00.022)       0:02:19.605 ********
2026-03-14 09:21:27.907087 | controller | skipping: [controller] => (item=openstack)
2026-03-14 09:21:27.936278 | controller | skipping: [controller] => (item=openstack-operators)
2026-03-14 09:21:27.936317 | controller | skipping: [controller]
2026-03-14 09:21:27.936326 | controller |
2026-03-14 09:21:27.936333 | controller | TASK [openshift_setup : Wait for the image registry to be ready kind=Deployment, name=image-registry, namespace=openshift-image-registry, kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, wait=True, wait_sleep=10, wait_timeout=600, wait_condition={'type': 'Available', 'status': 'True'}] ***
2026-03-14 09:21:27.936342 | controller | Saturday 14 March 2026  09:21:27 +0000 (0:00:00.029)       0:02:19.636 ********
2026-03-14 09:21:27.936349 | controller | Saturday 14 March 2026  09:21:27 +0000 (0:00:00.029)       0:02:19.635 ********
2026-03-14 09:21:27.936363 | controller | skipping: [controller]
2026-03-14 09:21:27.970008 | controller |
2026-03-14 09:21:27.970048 | controller | TASK [openshift_setup : Login into OpenShift internal registry output_dir={{ cifmw_openshift_setup_basedir }}/artifacts, script=podman login -u {{ cifmw_openshift_user }} -p {{ cifmw_openshift_token }} {%- if cifmw_openshift_setup_skip_internal_registry_tls_verify|bool %} --tls-verify=false {%- endif %} {{ cifmw_openshift_setup_registry_default_route.resources[0].spec.host }}] ***
2026-03-14 09:21:27.970058 | controller | Saturday 14 March 2026  09:21:27 +0000 (0:00:00.029)       0:02:19.665 ********
2026-03-14 09:21:27.970065 | controller | Saturday 14 March 2026  09:21:27 +0000 (0:00:00.029)       0:02:19.664 ********
2026-03-14 09:21:27.970078 | controller | skipping: [controller]
2026-03-14 09:21:28.000244 | controller |
2026-03-14 09:21:28.000290 | controller | TASK [Ensure we have custom CA installed on host role=install_ca] **************
2026-03-14 09:21:28.000301 | controller | Saturday 14 March 2026  09:21:27 +0000 (0:00:00.033)       0:02:19.699 ********
2026-03-14 09:21:28.000309 | controller | Saturday 14 March 2026  09:21:27 +0000 (0:00:00.033)       0:02:19.697 ********
2026-03-14 09:21:28.000332 | controller | skipping: [controller]
2026-03-14 09:21:28.030153 | controller |
2026-03-14 09:21:28.030194 | controller | TASK [openshift_setup : Update ca bundle _raw_params=update-ca-trust extract] ***
2026-03-14 09:21:28.030203 | controller | Saturday 14 March 2026  09:21:27 +0000 (0:00:00.030)       0:02:19.729 ********
2026-03-14 09:21:28.030209 | controller | Saturday 14 March 2026  09:21:27 +0000 (0:00:00.030)       0:02:19.728 ********
2026-03-14 09:21:28.030222 | controller | skipping: [controller]
2026-03-14 09:21:28.061557 | controller |
2026-03-14 09:21:28.061602 | controller | TASK [openshift_setup : Slurp CAs file src={{ cifmw_openshift_setup_ca_bundle_path }}] ***
2026-03-14 09:21:28.061631 | controller | Saturday 14 March 2026  09:21:28 +0000 (0:00:00.029)       0:02:19.759 ********
2026-03-14 09:21:28.061641 | controller | Saturday 14 March 2026  09:21:28 +0000 (0:00:00.029)       0:02:19.758 ********
2026-03-14 09:21:28.061653 | controller | skipping: [controller]
2026-03-14 09:21:28.093951 | controller |
2026-03-14 09:21:28.093991 | controller | TASK [openshift_setup : Create config map with registry CAs kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, definition={'apiVersion': 'v1', 'kind': 'ConfigMap', 'metadata': {'namespace': 'openshift-config', 'name': 'registry-cas'}, 'data': '{{ _config_map_data | items2dict  }}'}] ***
2026-03-14 09:21:28.094000 | controller | Saturday 14 March 2026  09:21:28 +0000 (0:00:00.031)       0:02:19.791 ********
2026-03-14 09:21:28.094007 | controller | Saturday 14 March 2026  09:21:28 +0000 (0:00:00.031)       0:02:19.789 ********
2026-03-14 09:21:28.094019 | controller | skipping: [controller]
2026-03-14 09:21:28.123193 | controller |
2026-03-14 09:21:28.123235 | controller | TASK [openshift_setup : Install Red Hat CA for pulling images from internal registry kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, merge_type=merge, definition={'apiVersion': 'config.openshift.io/v1', 'kind': 'Image', 'metadata': {'name': 'cluster'}, 'spec': {'additionalTrustedCA': {'name': 'registry-cas'}}}] ***
2026-03-14 09:21:28.123248 | controller | Saturday 14 March 2026  09:21:28 +0000 (0:00:00.032)       0:02:19.823 ********
2026-03-14 09:21:28.123255 | controller | Saturday 14 March 2026  09:21:28 +0000 (0:00:00.032)       0:02:19.822 ********
2026-03-14 09:21:28.123268 | controller | skipping: [controller]
2026-03-14 09:21:28.152520 | controller |
2026-03-14 09:21:28.152557 | controller | TASK [openshift_setup : Add insecure registry kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, merge_type=merge, definition={'apiVersion': 'config.openshift.io/v1', 'kind': 'Image', 'metadata': {'name': 'cluster'}, 'spec': {'registrySources': {'insecureRegistries': ['{{ cifmw_update_containers_registry }}'], 'allowedRegistries': '{{ all_registries }}'}}}] ***
2026-03-14 09:21:28.152567 | controller | Saturday 14 March 2026  09:21:28 +0000 (0:00:00.029)       0:02:19.852 ********
2026-03-14 09:21:28.152574 | controller | Saturday 14 March 2026  09:21:28 +0000 (0:00:00.029)       0:02:19.851 ********
2026-03-14 09:21:28.152586 | controller | skipping: [controller]
2026-03-14 09:21:28.184553 | controller |
2026-03-14 09:21:28.184589 | controller | TASK [openshift_setup : Create a ICSP with repository digest mirrors kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, definition={'apiVersion': 'operator.openshift.io/v1alpha1', 'kind': 'ImageContentSourcePolicy', 'metadata': {'name': 'registry-digest-mirrors'}, 'spec': {'repositoryDigestMirrors': '{{ cifmw_openshift_setup_digest_mirrors }}'}}] ***
2026-03-14 09:21:28.184596 | controller | Saturday 14 March 2026  09:21:28 +0000 (0:00:00.029)       0:02:19.882 ********
2026-03-14 09:21:28.184605 | controller | Saturday 14 March 2026  09:21:28 +0000 (0:00:00.029)       0:02:19.880 ********
2026-03-14 09:21:28.184626 | controller | skipping: [controller]
2026-03-14 09:21:29.174550 | controller |
2026-03-14 09:21:29.174587 | controller | TASK [openshift_setup : Gather network.operator info kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, api_version=operator.openshift.io/v1, kind=Network, name=cluster] ***
2026-03-14 09:21:29.174605 | controller | Saturday 14 March 2026  09:21:28 +0000 (0:00:00.032)       0:02:19.914 ********
2026-03-14 09:21:29.174637 | controller | Saturday 14 March 2026  09:21:28 +0000 (0:00:00.032)       0:02:19.912 ********
2026-03-14 09:21:29.174652 | controller | ok: [controller]
2026-03-14 09:21:30.113942 | controller |
2026-03-14 09:21:30.113978 | controller | TASK [openshift_setup : Patch network operator api_version=operator.openshift.io/v1, kubeconfig={{ cifmw_openshift_kubeconfig }}, kind=Network, name=cluster, persist_config=True, patch=[{'path': '/spec/defaultNetwork/ovnKubernetesConfig/gatewayConfig/routingViaHost', 'value': True, 'op': 'replace'}, {'path': '/spec/defaultNetwork/ovnKubernetesConfig/gatewayConfig/ipForwarding', 'value': 'Global', 'op': 'replace'}]] ***
2026-03-14 09:21:30.113985 | controller | Saturday 14 March 2026  09:21:29 +0000 (0:00:00.989)       0:02:20.904 ********
2026-03-14 09:21:30.113991 | controller | Saturday 14 March 2026  09:21:29 +0000 (0:00:00.989)       0:02:20.902 ********
2026-03-14 09:21:30.114000 | controller | changed: [controller]
2026-03-14 09:21:30.927429 | controller |
2026-03-14 09:21:30.927518 | controller | TASK [openshift_setup : Patch samples registry configuration kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, api_version=samples.operator.openshift.io/v1, kind=Config, name=cluster, patch=[{'op': 'replace', 'path': '/spec/samplesRegistry', 'value': '{{ cifmw_openshift_setup_samples_registry }}'}]] ***
2026-03-14 09:21:30.927542 | controller | Saturday 14 March 2026  09:21:30 +0000 (0:00:00.939)       0:02:21.843 ********
2026-03-14 09:21:30.927558 | controller | Saturday 14 March 2026  09:21:30 +0000 (0:00:00.939)       0:02:21.842 ********
2026-03-14 09:21:30.927579 | controller | changed: [controller]
2026-03-14 09:21:30.956402 | controller |
2026-03-14 09:21:30.956449 | controller | TASK [openshift_setup : Delete the pods from openshift-marketplace namespace kind=Pod, state=absent, delete_all=True, kubeconfig={{ cifmw_openshift_kubeconfig }}, namespace=openshift-marketplace] ***
2026-03-14 09:21:30.956458 | controller | Saturday 14 March 2026  09:21:30 +0000 (0:00:00.812)       0:02:22.656 ********
2026-03-14 09:21:30.956467 | controller | Saturday 14 March 2026  09:21:30 +0000 (0:00:00.812)       0:02:22.654 ********
2026-03-14 09:21:30.956483 | controller | skipping: [controller]
2026-03-14 09:21:30.991015 | controller |
2026-03-14 09:21:30.991064 | controller | TASK [openshift_setup : Wait for openshift-marketplace pods to be running _raw_params=oc wait pod --all --for=condition=Ready -n openshift-marketplace --timeout=1m] ***
2026-03-14 09:21:30.991077 | controller | Saturday 14 March 2026  09:21:30 +0000 (0:00:00.029)       0:02:22.685 ********
2026-03-14 09:21:30.991086 | controller | Saturday 14 March 2026  09:21:30 +0000 (0:00:00.029)       0:02:22.684 ********
2026-03-14 09:21:30.991101 | controller | skipping: [controller]
2026-03-14 09:21:31.078660 | controller |
2026-03-14 09:21:31.078692 | controller | TASK [Deploy Observability operator. name=openshift_obs] ***********************
2026-03-14 09:21:31.078698 | controller | Saturday 14 March 2026  09:21:30 +0000 (0:00:00.034)       0:02:22.720 ********
2026-03-14 09:21:31.078703 | controller | Saturday 14 March 2026  09:21:30 +0000 (0:00:00.034)       0:02:22.718 ********
2026-03-14 09:21:31.078711 | controller |
2026-03-14 09:21:31.851797 | controller | TASK [openshift_obs : Install cluster observability operator. definition={{cifmw_openshift_obs_definition }}, kubeconfig={{ cifmw_openshift_kubeconfig }}, state=present] ***
2026-03-14 09:21:31.851870 | controller | Saturday 14 March 2026  09:21:31 +0000 (0:00:00.088)       0:02:22.808 ********
2026-03-14 09:21:31.851885 | controller | Saturday 14 March 2026  09:21:31 +0000 (0:00:00.088)       0:02:22.806 ********
2026-03-14 09:21:31.851905 | controller | changed: [controller]
2026-03-14 09:22:22.948624 | controller |
2026-03-14 09:22:22.948676 | controller | TASK [openshift_obs : Wait for observability operator deployment kind=Deployment, namespace=openshift-operators, name=observability-operator, wait=True, wait_timeout=300, wait_condition={'type': 'Available', 'status': 'True'}, kubeconfig={{ cifmw_openshift_kubeconfig }}] ***
2026-03-14 09:22:22.948694 | controller | Saturday 14 March 2026  09:21:31 +0000 (0:00:00.773)       0:02:23.581 ********
2026-03-14 09:22:22.948700 | controller | Saturday 14 March 2026  09:21:31 +0000 (0:00:00.773)       0:02:23.579 ********
2026-03-14 09:22:22.948711 | controller | ok: [controller]
2026-03-14 09:22:23.815904 | controller |
2026-03-14 09:22:23.815939 | controller | TASK [openshift_obs : Wait for observability-operator pod kind=Pod, namespace=openshift-operators, label_selectors=['app.kubernetes.io/name = observability-operator'], wait=True, wait_timeout=300, wait_condition={'type': 'Ready', 'status': 'True'}, kubeconfig={{ cifmw_openshift_kubeconfig }}] ***
2026-03-14 09:22:23.815946 | controller | Saturday 14 March 2026  09:22:22 +0000 (0:00:51.096)       0:03:14.678 ********
2026-03-14 09:22:23.815951 | controller | Saturday 14 March 2026  09:22:22 +0000 (0:00:51.096)       0:03:14.676 ********
2026-03-14 09:22:23.815960 | controller | ok: [controller]
2026-03-14 09:22:23.841886 | controller |
2026-03-14 09:22:23.841960 | controller | TASK [Deploy Metal3 BMHs name=deploy_bmh] **************************************
2026-03-14 09:22:23.841973 | controller | Saturday 14 March 2026  09:22:23 +0000 (0:00:00.867)       0:03:15.545 ********
2026-03-14 09:22:23.841988 | controller | Saturday 14 March 2026  09:22:23 +0000 (0:00:00.867)       0:03:15.544 ********
2026-03-14 09:22:23.842010 | controller | skipping: [controller]
2026-03-14 09:22:23.876720 | controller |
2026-03-14 09:22:23.876792 | controller | TASK [Install certmanager operator role name=cert_manager] *********************
2026-03-14 09:22:23.876801 | controller | Saturday 14 March 2026  09:22:23 +0000 (0:00:00.025)       0:03:15.571 ********
2026-03-14 09:22:23.876808 | controller | Saturday 14 March 2026  09:22:23 +0000 (0:00:00.025)       0:03:15.569 ********
2026-03-14 09:22:23.876825 | controller | skipping: [controller]
2026-03-14 09:22:23.908341 | controller |
2026-03-14 09:22:23.908404 | controller | TASK [Configure hosts networking using nmstate name=ci_nmstate] ****************
2026-03-14 09:22:23.908413 | controller | Saturday 14 March 2026  09:22:23 +0000 (0:00:00.034)       0:03:15.606 ********
2026-03-14 09:22:23.908419 | controller | Saturday 14 March 2026  09:22:23 +0000 (0:00:00.034)       0:03:15.604 ********
2026-03-14 09:22:23.908435 | controller | skipping: [controller]
2026-03-14 09:22:23.934222 | controller |
2026-03-14 09:22:23.934278 | controller | TASK [Configure multus networks name=ci_multus] ********************************
2026-03-14 09:22:23.934287 | controller | Saturday 14 March 2026  09:22:23 +0000 (0:00:00.031)       0:03:15.637 ********
2026-03-14 09:22:23.934294 | controller | Saturday 14 March 2026  09:22:23 +0000 (0:00:00.031)       0:03:15.636 ********
2026-03-14 09:22:23.934308 | controller | skipping: [controller]
2026-03-14 09:22:23.959246 | controller |
2026-03-14 09:22:23.959307 | controller | TASK [Deploy Sushy Emulator service pod name=sushy_emulator] *******************
2026-03-14 09:22:23.959316 | controller | Saturday 14 March 2026  09:22:23 +0000 (0:00:00.026)       0:03:15.664 ********
2026-03-14 09:22:23.959322 | controller | Saturday 14 March 2026  09:22:23 +0000 (0:00:00.026)       0:03:15.662 ********
2026-03-14 09:22:23.959337 | controller | skipping: [controller]
2026-03-14 09:22:23.984547 | controller |
2026-03-14 09:22:23.984631 | controller | TASK [Setup Libvirt on controller name=libvirt_manager] ************************
2026-03-14 09:22:23.984642 | controller | Saturday 14 March 2026  09:22:23 +0000 (0:00:00.024)       0:03:15.688 ********
2026-03-14 09:22:23.984649 | controller | Saturday 14 March 2026  09:22:23 +0000 (0:00:00.024)       0:03:15.687 ********
2026-03-14 09:22:23.984664 | controller | skipping: [controller]
2026-03-14 09:22:24.011376 | controller |
2026-03-14 09:22:24.011436 | controller | TASK [Prepare container package builder name=pkg_build] ************************
2026-03-14 09:22:24.011444 | controller | Saturday 14 March 2026  09:22:23 +0000 (0:00:00.025)       0:03:15.714 ********
2026-03-14 09:22:24.011451 | controller | Saturday 14 March 2026  09:22:23 +0000 (0:00:00.025)       0:03:15.712 ********
2026-03-14 09:22:24.011466 | controller | skipping: [controller]
2026-03-14 09:22:24.078231 | controller |
2026-03-14 09:22:24.078287 | controller | TASK [run_hook : Assert parameters are valid quiet=True, that=['_list_hooks is not string', '_list_hooks is not mapping', '_list_hooks is iterable', '(hooks | default([])) is not string', '(hooks | default([])) is not mapping', '(hooks | default([])) is iterable']] ***
2026-03-14 09:22:24.078302 | controller | Saturday 14 March 2026  09:22:24 +0000 (0:00:00.027)       0:03:15.741 ********
2026-03-14 09:22:24.078308 | controller | Saturday 14 March 2026  09:22:24 +0000 (0:00:00.027)       0:03:15.739 ********
2026-03-14 09:22:24.078322 | controller | ok: [controller]
2026-03-14 09:22:24.182978 | controller |
2026-03-14 09:22:24.183027 | controller | TASK [run_hook : Assert single hooks are all mappings quiet=True, that=['_not_mapping_hooks | length == 0'], msg=All single hooks must be a list of mappings or a mapping.] ***
2026-03-14 09:22:24.183035 | controller | Saturday 14 March 2026  09:22:24 +0000 (0:00:00.066)       0:03:15.807 ********
2026-03-14 09:22:24.183039 | controller | Saturday 14 March 2026  09:22:24 +0000 (0:00:00.066)       0:03:15.806 ********
2026-03-14 09:22:24.183050 | controller | ok: [controller]
2026-03-14 09:22:24.290388 | controller |
2026-03-14 09:22:24.290443 | controller | TASK [run_hook : Loop on hooks for post_infra _raw_params={{ hook.type }}.yml] ***
2026-03-14 09:22:24.290450 | controller | Saturday 14 March 2026  09:22:24 +0000 (0:00:00.104)       0:03:15.912 ********
2026-03-14 09:22:24.290455 | controller | Saturday 14 March 2026  09:22:24 +0000 (0:00:00.104)       0:03:15.911 ********
2026-03-14 09:22:24.290467 | controller | skipping: [controller]
2026-03-14 09:22:24.357815 | controller |
2026-03-14 09:22:24.357878 | controller | TASK [Load parameters dir={{ item }}] ******************************************
2026-03-14 09:22:24.357886 | controller | Saturday 14 March 2026  09:22:24 +0000 (0:00:00.107)       0:03:16.020 ********
2026-03-14 09:22:24.357891 | controller | Saturday 14 March 2026  09:22:24 +0000 (0:00:00.107)       0:03:16.018 ********
2026-03-14 09:22:24.357905 | controller | ok: [controller] => (item=/home/zuul/ci-framework-data/artifacts/parameters)
2026-03-14 09:22:24.394401 | controller | ok: [controller] => (item=/etc/ci/env)
2026-03-14 09:22:24.394456 | controller |
2026-03-14 09:22:24.394464 | controller | TASK [Ensure that the isolated net was configured for crc that=['crc_ci_bootstrap_networks_out is defined', 'crc_ci_bootstrap_networks_out[_crc_hostname] is defined', "crc_ci_bootstrap_networks_out[_crc_hostname]['default'] is defined"]] ***
2026-03-14 09:22:24.394476 | controller | Saturday 14 March 2026  09:22:24 +0000 (0:00:00.066)       0:03:16.086 ********
2026-03-14 09:22:24.394481 | controller | Saturday 14 March 2026  09:22:24 +0000 (0:00:00.066)       0:03:16.085 ********
2026-03-14 09:22:24.394491 | controller | ok: [controller] =>
2026-03-14 09:22:24.460915 | controller |     changed: false
2026-03-14 09:22:24.460968 | controller |     msg: All assertions passed
2026-03-14 09:22:24.460975 | controller |
2026-03-14 09:22:24.460981 | controller | TASK [Set facts for further usage within the framework cifmw_edpm_prepare_extra_vars={'NNCP_INTERFACE': '{{ crc_ci_bootstrap_networks_out.crc.default.iface }}', 'NETWORK_MTU': '{{ crc_ci_bootstrap_networks_out.crc.default.mtu }}', 'NNCP_DNS_SERVER': "{{\n  cifmw_nncp_dns_server |\n  default(crc_ci_bootstrap_networks_out[_crc_hostname].default.ip) |\n  split('/') | first\n}}"}] ***
2026-03-14 09:22:24.460988 | controller | Saturday 14 March 2026  09:22:24 +0000 (0:00:00.037)       0:03:16.123 ********
2026-03-14 09:22:24.460993 | controller | Saturday 14 March 2026  09:22:24 +0000 (0:00:00.037)       0:03:16.122 ********
2026-03-14 09:22:24.461004 | controller | ok: [controller]
2026-03-14 09:22:24.516889 | controller |
2026-03-14 09:22:24.516942 | controller | PLAY [Deploy Openstack Operators] **********************************************
2026-03-14 09:22:24.516949 | controller |
2026-03-14 09:22:24.516954 | controller | TASK [Use the locally built operators if any _local_operators_indexes={{
2026-03-14 09:22:24.516958 | controller |   _local_operators_indexes|default({}) |
2026-03-14 09:22:24.516964 | controller |   combine({ item.key.split('-')[0]|upper+'_IMG':
2026-03-14 09:22:24.516968 | controller |   cifmw_operator_build_output['operators'][item.key].image_catalog})
2026-03-14 09:22:24.516973 | controller | }}] ***
2026-03-14 09:22:24.516978 | controller | Saturday 14 March 2026  09:22:24 +0000 (0:00:00.066)       0:03:16.190 ********
2026-03-14 09:22:24.516983 | controller | Saturday 14 March 2026  09:22:24 +0000 (0:00:00.066)       0:03:16.189 ********
2026-03-14 09:22:24.516997 | controller | ok: [controller] => (item={'key': 'openstack-operator', 'value': {'git_commit_hash': '63019d9ad483a325f5e70bb2c5e3438e3468b31f', 'git_src_dir': '/home/zuul-worker/src/github.com/openstack-k8s-operators/openstack-operator', 'image': '38.102.83.243:5001/openstack-k8s-operators/openstack-operator:63019d9ad483a325f5e70bb2c5e3438e3468b31f', 'image_bundle': '38.102.83.243:5001/openstack-k8s-operators/openstack-operator-bundle:63019d9ad483a325f5e70bb2c5e3438e3468b31f', 'image_catalog': '38.102.83.243:5001/openstack-k8s-operators/openstack-operator-index:63019d9ad483a325f5e70bb2c5e3438e3468b31f'}})
2026-03-14 09:22:24.545327 | controller | ok: [controller] => (item={'key': 'watcher-operator', 'value': {'git_commit_hash': '0d9467a022a6d6a8b6465cdbc9e4aefc533e796a', 'git_src_dir': '/home/zuul-worker/src/github.com/openstack-k8s-operators/watcher-operator', 'image': '38.102.83.243:5001/openstack-k8s-operators/watcher-operator:0d9467a022a6d6a8b6465cdbc9e4aefc533e796a', 'image_bundle': '38.102.83.243:5001/openstack-k8s-operators/watcher-operator-bundle:0d9467a022a6d6a8b6465cdbc9e4aefc533e796a', 'image_catalog': '38.102.83.243:5001/openstack-k8s-operators/watcher-operator-index:0d9467a022a6d6a8b6465cdbc9e4aefc533e796a'}})
2026-03-14 09:22:24.545382 | controller |
2026-03-14 09:22:24.545389 | controller | TASK [Set install_yamls Makefile environment variables cifmw_edpm_prepare_common_env={{
2026-03-14 09:22:24.545395 | controller |   cifmw_install_yamls_environment |
2026-03-14 09:22:24.545400 | controller |   combine({'PATH': cifmw_path}) |
2026-03-14 09:22:24.545405 | controller |   combine(cifmw_edpm_prepare_extra_vars | default({}))
2026-03-14 09:22:24.545410 | controller | }}, cifmw_edpm_prepare_operators_build_output={{ operators_build_output }}, cifmw_edpm_prepare_make_openstack_env={{ _local_operators_indexes | combine(_openstack_operator_images) }}] ***
2026-03-14 09:22:24.545416 | controller | Saturday 14 March 2026  09:22:24 +0000 (0:00:00.055)       0:03:16.245 ********
2026-03-14 09:22:24.545421 | controller | Saturday 14 March 2026  09:22:24 +0000 (0:00:00.055)       0:03:16.244 ********
2026-03-14 09:22:24.545432 | controller | ok: [controller]
2026-03-14 09:22:24.981629 | controller |
2026-03-14 09:22:24.981663 | controller | TASK [detect if openstack operator is installed _raw_params=oc get sub --ignore-not-found=true -n openstack-operators -o name openstack-operator] ***
2026-03-14 09:22:24.981671 | controller | Saturday 14 March 2026  09:22:24 +0000 (0:00:00.029)       0:03:16.275 ********
2026-03-14 09:22:24.981676 | controller | Saturday 14 March 2026  09:22:24 +0000 (0:00:00.029)       0:03:16.273 ********
2026-03-14 09:22:24.981687 | controller | changed: [controller]
2026-03-14 09:22:25.014585 | controller |
2026-03-14 09:22:25.014645 | controller | TASK [Install openstack operator and wait for the csv to succeed name=install_yamls_makes, tasks_from=make_openstack_init] ***
2026-03-14 09:22:25.014652 | controller | Saturday 14 March 2026  09:22:24 +0000 (0:00:00.436)       0:03:16.711 ********
2026-03-14 09:22:25.014657 | controller | Saturday 14 March 2026  09:22:24 +0000 (0:00:00.436)       0:03:16.709 ********
2026-03-14 09:22:25.014667 | controller |
2026-03-14 09:22:25.051780 | controller | TASK [install_yamls_makes : Debug make_openstack_init_env var=make_openstack_init_env] ***
2026-03-14 09:22:25.051816 | controller | Saturday 14 March 2026  09:22:25 +0000 (0:00:00.032)       0:03:16.744 ********
2026-03-14 09:22:25.051823 | controller | Saturday 14 March 2026  09:22:25 +0000 (0:00:00.032)       0:03:16.742 ********
2026-03-14 09:22:25.051833 | controller | ok: [controller] =>
2026-03-14 09:22:25.078829 | controller |     make_openstack_init_env:
2026-03-14 09:22:25.078850 | controller |       BMO_SETUP: false
2026-03-14 09:22:25.078856 | controller |       CHECKOUT_FROM_OPENSTACK_REF: 'true'
2026-03-14 09:22:25.078861 | controller |       KUBECONFIG: /home/zuul/.crc/machines/crc/kubeconfig
2026-03-14 09:22:25.078866 | controller |       NETWORK_MTU: 1500
2026-03-14 09:22:25.078870 | controller |       NNCP_DNS_SERVER: 192.168.122.10
2026-03-14 09:22:25.078879 | controller |       NNCP_INTERFACE: ens7
2026-03-14 09:22:25.078884 | controller |       OPENSTACK_BUNDLE_IMG: 38.102.83.243:5001/openstack-k8s-operators/openstack-operator-bundle:63019d9ad483a325f5e70bb2c5e3438e3468b31f
2026-03-14 09:22:25.078890 | controller |       OPENSTACK_IMG: 38.102.83.243:5001/openstack-k8s-operators/openstack-operator-index:63019d9ad483a325f5e70bb2c5e3438e3468b31f
2026-03-14 09:22:25.078895 | controller |       OPENSTACK_K8S_BRANCH: main
2026-03-14 09:22:25.078904 | controller |       OUT: /home/zuul/ci-framework-data/artifacts/manifests
2026-03-14 09:22:25.078908 | controller |       OUTPUT_DIR: /home/zuul/ci-framework-data/artifacts/edpm
2026-03-14 09:22:25.078913 | controller |       PATH: /home/zuul/.crc/bin:/home/zuul/.crc/bin/oc:/home/zuul/bin:/home/zuul/.local/bin:/home/zuul/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
2026-03-14 09:22:25.078918 | controller |       WATCHER_BRANCH: ''
2026-03-14 09:22:25.078923 | controller |       WATCHER_IMG: 38.102.83.243:5001/openstack-k8s-operators/watcher-operator-index:0d9467a022a6d6a8b6465cdbc9e4aefc533e796a
2026-03-14 09:22:25.078927 | controller |       WATCHER_REPO: /home/zuul/src/github.com/openstack-k8s-operators/watcher-operator
2026-03-14 09:22:25.078932 | controller |
2026-03-14 09:22:25.078936 | controller | TASK [install_yamls_makes : Debug make_openstack_init_params var=make_openstack_init_params] ***
2026-03-14 09:22:25.078941 | controller | Saturday 14 March 2026  09:22:25 +0000 (0:00:00.037)       0:03:16.781 ********
2026-03-14 09:22:25.078945 | controller | Saturday 14 March 2026  09:22:25 +0000 (0:00:00.037)       0:03:16.779 ********
2026-03-14 09:22:25.078953 | controller | skipping: [controller]
2026-03-14 09:27:41.711385 | controller |
2026-03-14 09:27:41.711422 | controller | TASK [install_yamls_makes : Run openstack_init output_dir={{ cifmw_basedir }}/artifacts, chdir=/home/zuul/src/github.com/openstack-k8s-operators/install_yamls, script=make openstack_init, dry_run={{ make_openstack_init_dryrun|default(false)|bool }}, extra_args={{ dict((make_openstack_init_env|default({})), **(make_openstack_init_params|default({}))) }}] ***
2026-03-14 09:27:41.711431 | controller | Saturday 14 March 2026  09:22:25 +0000 (0:00:00.027)       0:03:16.808 ********
2026-03-14 09:27:41.711438 | controller | Saturday 14 March 2026  09:22:25 +0000 (0:00:00.027)       0:03:16.806 ********
2026-03-14 09:27:41.711449 | controller | [WARNING]: conditional statements should not include jinja2 templating
2026-03-14 09:27:41.731557 | controller | delimiters such as {{ }} or {% %}. Found: {{ make_openstack_init_until |
2026-03-14 09:27:41.731590 | controller | default(true) }}
2026-03-14 09:27:41.731630 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_003_run_openstack.log
2026-03-14 09:27:41.766214 | controller | changed: [controller]
2026-03-14 09:27:41.766245 | controller |
2026-03-14 09:27:41.766251 | controller | TASK [Run hooks after installing openstack name=run_hook] **********************
2026-03-14 09:27:41.766256 | controller | Saturday 14 March 2026  09:27:41 +0000 (0:05:16.652)       0:08:33.461 ********
2026-03-14 09:27:41.766260 | controller | Saturday 14 March 2026  09:27:41 +0000 (0:05:16.652)       0:08:33.459 ********
2026-03-14 09:27:41.766269 | controller |
2026-03-14 09:27:41.865544 | controller | TASK [run_hook : Assert parameters are valid quiet=True, that=['_list_hooks is not string', '_list_hooks is not mapping', '_list_hooks is iterable', '(hooks | default([])) is not string', '(hooks | default([])) is not mapping', '(hooks | default([])) is iterable']] ***
2026-03-14 09:27:41.865678 | controller | Saturday 14 March 2026  09:27:41 +0000 (0:00:00.034)       0:08:33.496 ********
2026-03-14 09:27:41.865693 | controller | Saturday 14 March 2026  09:27:41 +0000 (0:00:00.034)       0:08:33.494 ********
2026-03-14 09:27:41.865711 | controller | ok: [controller]
2026-03-14 09:27:41.967113 | controller |
2026-03-14 09:27:41.967151 | controller | TASK [run_hook : Assert single hooks are all mappings quiet=True, that=['_not_mapping_hooks | length == 0'], msg=All single hooks must be a list of mappings or a mapping.] ***
2026-03-14 09:27:41.967158 | controller | Saturday 14 March 2026  09:27:41 +0000 (0:00:00.098)       0:08:33.594 ********
2026-03-14 09:27:41.967164 | controller | Saturday 14 March 2026  09:27:41 +0000 (0:00:00.098)       0:08:33.593 ********
2026-03-14 09:27:41.967174 | controller | ok: [controller]
2026-03-14 09:27:42.130286 | controller |
2026-03-14 09:27:42.130329 | controller | TASK [run_hook : Loop on hooks for post_install_operators_kuttl_from_operator _raw_params={{ hook.type }}.yml] ***
2026-03-14 09:27:42.130338 | controller | Saturday 14 March 2026  09:27:41 +0000 (0:00:00.102)       0:08:33.696 ********
2026-03-14 09:27:42.130345 | controller | Saturday 14 March 2026  09:27:41 +0000 (0:00:00.102)       0:08:33.695 ********
2026-03-14 09:27:42.130359 | controller | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/run_hook/tasks/playbook.yml for controller => (item={'name': 'Deploy watcher operator', 'type': 'playbook', 'source': '/home/zuul/src/github.com/openstack-k8s-operators/watcher-operator/ci/playbooks/deploy_watcher_operator.yaml', 'extra_vars': {'content_provider_os_registry_url': '38.102.83.243:5001/podified-master-centos10', 'watcher_catalog_image': '38.102.83.243:5001/openstack-k8s-operators/watcher-operator-index:0d9467a022a6d6a8b6465cdbc9e4aefc533e796a', 'watcher_services_tag': 'watcher_latest', 'watcher_repo': '/home/zuul/src/github.com/openstack-k8s-operators/watcher-operator'}})
2026-03-14 09:27:42.190649 | controller |
2026-03-14 09:27:42.190684 | controller | TASK [run_hook : Set playbook path for Deploy watcher operator cifmw_basedir={{ _bdir }}, hook_name={{ _hook_name }}, playbook_path={{ _play | realpath }}, log_path={{ _bdir }}/logs/{{ step }}_{{ _hook_name }}.log, extra_vars=-e namespace={{ cifmw_openstack_namespace }} {%- if hook.extra_vars is defined and hook.extra_vars|length > 0 -%} {% for key,value in hook.extra_vars.items() -%} {%- if key == 'file' %} -e "@{{ value }}" {%- else %} -e "{{ key }}={{ value }}" {%- endif %} {%- endfor %} {%- endif %}] ***
2026-03-14 09:27:42.190691 | controller | Saturday 14 March 2026  09:27:42 +0000 (0:00:00.162)       0:08:33.859 ********
2026-03-14 09:27:42.190699 | controller | Saturday 14 March 2026  09:27:42 +0000 (0:00:00.162)       0:08:33.857 ********
2026-03-14 09:27:42.190708 | controller | ok: [controller]
2026-03-14 09:27:42.441166 | controller |
2026-03-14 09:27:42.441221 | controller | TASK [run_hook : Get file stat path={{ playbook_path }}] ***********************
2026-03-14 09:27:42.441235 | controller | Saturday 14 March 2026  09:27:42 +0000 (0:00:00.061)       0:08:33.920 ********
2026-03-14 09:27:42.441245 | controller | Saturday 14 March 2026  09:27:42 +0000 (0:00:00.061)       0:08:33.918 ********
2026-03-14 09:27:42.441262 | controller | ok: [controller]
2026-03-14 09:27:42.471941 | controller |
2026-03-14 09:27:42.471994 | controller | TASK [run_hook : Fail if playbook doesn't exist msg=Playbook {{ playbook_path }} doesn't seem to exist.] ***
2026-03-14 09:27:42.472007 | controller | Saturday 14 March 2026  09:27:42 +0000 (0:00:00.249)       0:08:34.170 ********
2026-03-14 09:27:42.472017 | controller | Saturday 14 March 2026  09:27:42 +0000 (0:00:00.249)       0:08:34.168 ********
2026-03-14 09:27:42.472035 | controller | skipping: [controller]
2026-03-14 09:27:42.703678 | controller |
2026-03-14 09:27:42.703717 | controller | TASK [run_hook : Get parameters files paths={{
2026-03-14 09:27:42.703726 | controller |   (cifmw_basedir, 'artifacts/parameters') | path_join
2026-03-14 09:27:42.703732 | controller | }}, file_type=file, patterns=*.yml] ***
2026-03-14 09:27:42.703738 | controller | Saturday 14 March 2026  09:27:42 +0000 (0:00:00.030)       0:08:34.200 ********
2026-03-14 09:27:42.703745 | controller | Saturday 14 March 2026  09:27:42 +0000 (0:00:00.030)       0:08:34.199 ********
2026-03-14 09:27:42.703756 | controller | ok: [controller]
2026-03-14 09:27:42.731058 | controller |
2026-03-14 09:27:42.731110 | controller | TASK [run_hook : Add parameters artifacts as extra variables extra_vars={{ extra_vars }} {% for file in cifmw_run_hook_parameters_files.files %} -e "@{{ file.path }}" {%- endfor %}] ***
2026-03-14 09:27:42.731121 | controller | Saturday 14 March 2026  09:27:42 +0000 (0:00:00.232)       0:08:34.433 ********
2026-03-14 09:27:42.731128 | controller | Saturday 14 March 2026  09:27:42 +0000 (0:00:00.232)       0:08:34.431 ********
2026-03-14 09:27:42.731140 | controller | ok: [controller]
2026-03-14 09:27:43.204636 | controller |
2026-03-14 09:27:43.204732 | controller | TASK [run_hook : Ensure log directory exists path={{ log_path | dirname }}, state=directory, mode=0755] ***
2026-03-14 09:27:43.204741 | controller | Saturday 14 March 2026  09:27:42 +0000 (0:00:00.027)       0:08:34.460 ********
2026-03-14 09:27:43.204747 | controller | Saturday 14 March 2026  09:27:42 +0000 (0:00:00.027)       0:08:34.459 ********
2026-03-14 09:27:43.204759 | controller | ok: [controller]
2026-03-14 09:27:43.425978 | controller |
2026-03-14 09:27:43.426048 | controller | TASK [run_hook : Ensure artifacts directory exists path={{ cifmw_basedir }}/artifacts, state=directory, mode=0755] ***
2026-03-14 09:27:43.426062 | controller | Saturday 14 March 2026  09:27:43 +0000 (0:00:00.472)       0:08:34.933 ********
2026-03-14 09:27:43.426072 | controller | Saturday 14 March 2026  09:27:43 +0000 (0:00:00.472)       0:08:34.932 ********
2026-03-14 09:27:43.426100 | controller | ok: [controller]
2026-03-14 09:28:43.678698 | controller |
2026-03-14 09:28:43.678750 | controller | TASK [run_hook : Run hook without retry - Deploy watcher operator] *************
2026-03-14 09:28:43.678759 | controller | Saturday 14 March 2026  09:27:43 +0000 (0:00:00.221)       0:08:35.154 ********
2026-03-14 09:28:43.678765 | controller | Saturday 14 March 2026  09:27:43 +0000 (0:00:00.221)       0:08:35.153 ********
2026-03-14 09:28:43.678777 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_004_run_hook_without_retry_deploy.log
2026-03-14 09:28:43.700449 | controller | changed: [controller]
2026-03-14 09:28:43.700519 | controller |
2026-03-14 09:28:43.700528 | controller | TASK [run_hook : Run hook with retry - Deploy watcher operator] ****************
2026-03-14 09:28:43.700535 | controller | Saturday 14 March 2026  09:28:43 +0000 (0:01:00.252)       0:09:35.407 ********
2026-03-14 09:28:43.700541 | controller | Saturday 14 March 2026  09:28:43 +0000 (0:01:00.252)       0:09:35.406 ********
2026-03-14 09:28:43.700556 | controller | skipping: [controller]
2026-03-14 09:28:43.908225 | controller |
2026-03-14 09:28:43.908257 | controller | TASK [run_hook : Check if we have a file path={{ cifmw_basedir }}/artifacts/{{ step }}_{{ hook_name }}.yml] ***
2026-03-14 09:28:43.908264 | controller | Saturday 14 March 2026  09:28:43 +0000 (0:00:00.022)       0:09:35.430 ********
2026-03-14 09:28:43.908269 | controller | Saturday 14 March 2026  09:28:43 +0000 (0:00:00.022)       0:09:35.428 ********
2026-03-14 09:28:43.908278 | controller | ok: [controller]
2026-03-14 09:28:43.927483 | controller |
2026-03-14 09:28:43.927516 | controller | TASK [run_hook : Load generated content in main playbook file={{ cifmw_basedir }}/artifacts/{{ step }}_{{ hook_name }}.yml] ***
2026-03-14 09:28:43.927522 | controller | Saturday 14 March 2026  09:28:43 +0000 (0:00:00.207)       0:09:35.638 ********
2026-03-14 09:28:43.927527 | controller | Saturday 14 March 2026  09:28:43 +0000 (0:00:00.207)       0:09:35.636 ********
2026-03-14 09:28:43.927535 | controller | skipping: [controller]
2026-03-14 09:31:22.446583 | controller |
2026-03-14 09:31:22.446658 | controller | TASK [install kuttl test_suite dependencies chdir={{ ansible_user_dir }}/{{ operator_basedir }}, _raw_params=make kuttl-test-prep] ***
2026-03-14 09:31:22.446667 | controller | Saturday 14 March 2026  09:28:43 +0000 (0:00:00.019)       0:09:35.657 ********
2026-03-14 09:31:22.446674 | controller | Saturday 14 March 2026  09:28:43 +0000 (0:00:00.019)       0:09:35.655 ********
2026-03-14 09:31:22.446686 | controller | changed: [controller]
2026-03-14 09:31:22.521913 | controller |
2026-03-14 09:31:22.521972 | controller | PLAY RECAP *********************************************************************
2026-03-14 09:31:22.521979 | controller | controller                 : ok=117  changed=40   unreachable=0    failed=0    skipped=80   rescued=0    ignored=1
2026-03-14 09:31:22.521983 | controller |
2026-03-14 09:31:22.521988 | controller | Saturday 14 March 2026  09:31:22 +0000 (0:02:38.517)       0:12:14.174 ********
2026-03-14 09:31:22.521992 | controller | ===============================================================================
2026-03-14 09:31:22.521997 | controller | install_yamls_makes : Run openstack_init ------------------------------ 316.65s
2026-03-14 09:31:22.522001 | controller | install kuttl test_suite dependencies --------------------------------- 158.52s
2026-03-14 09:31:22.522008 | controller | run_hook : Run hook without retry - Deploy watcher operator ------------ 60.25s
2026-03-14 09:31:22.522013 | controller | openshift_obs : Wait for observability operator deployment ------------- 51.10s
2026-03-14 09:31:22.522017 | controller | install_yamls_makes : Run download_tools ------------------------------- 37.46s
2026-03-14 09:31:22.522022 | controller | run_hook : Run hook without retry - Download needed tools -------------- 34.15s
2026-03-14 09:31:22.522026 | controller | ci_setup : Install needed packages ------------------------------------- 27.78s
2026-03-14 09:31:22.522030 | controller | repo_setup : Initialize python venv and install requirements ------------ 7.74s
2026-03-14 09:31:22.522035 | controller | ci_setup : Install openshift client ------------------------------------- 5.47s
2026-03-14 09:31:22.522039 | controller | openshift_setup : Create required namespaces ---------------------------- 2.20s
2026-03-14 09:31:22.522043 | controller | repo_setup : Make sure git-core package is installed -------------------- 1.28s
2026-03-14 09:31:22.522057 | controller | Gathering Facts --------------------------------------------------------- 1.16s
2026-03-14 09:31:22.522061 | controller | ci_setup : Manage directories ------------------------------------------- 1.08s
2026-03-14 09:31:22.522066 | controller | openshift_setup : Gather network.operator info -------------------------- 0.99s
2026-03-14 09:31:22.522070 | controller | discover_latest_image : Get latest image for the current entry ---------- 0.97s
2026-03-14 09:31:22.522075 | controller | repo_setup : Get repo-setup repository ---------------------------------- 0.94s
2026-03-14 09:31:22.522079 | controller | openshift_setup : Patch network operator -------------------------------- 0.94s
2026-03-14 09:31:22.522083 | controller | repo_setup : Dump full hash in delorean.repo.md5 file ------------------- 0.89s
2026-03-14 09:31:22.522088 | controller | repo_setup : Install repo-setup package --------------------------------- 0.89s
2026-03-14 09:31:22.522092 | controller | openshift_obs : Wait for observability-operator pod --------------------- 0.87s
2026-03-14 09:31:22.522096 | controller | Saturday 14 March 2026  09:31:22 +0000 (0:02:38.518)       0:12:14.174 ********
2026-03-14 09:31:22.522100 | controller | ===============================================================================
2026-03-14 09:31:22.522104 | controller | install_yamls_makes --------------------------------------------------- 354.22s
2026-03-14 09:31:22.522109 | controller | ansible.builtin.command ----------------------------------------------- 158.96s
2026-03-14 09:31:22.522113 | controller | run_hook --------------------------------------------------------------- 98.07s
2026-03-14 09:31:22.522117 | controller | openshift_obs ---------------------------------------------------------- 52.74s
2026-03-14 09:31:22.522121 | controller | ci_setup --------------------------------------------------------------- 35.53s
2026-03-14 09:31:22.522126 | controller | repo_setup ------------------------------------------------------------- 16.67s
2026-03-14 09:31:22.522130 | controller | openshift_setup --------------------------------------------------------- 5.62s
2026-03-14 09:31:22.522134 | controller | openshift_login --------------------------------------------------------- 4.09s
2026-03-14 09:31:22.522138 | controller | install_yamls ----------------------------------------------------------- 3.37s
2026-03-14 09:31:22.522143 | controller | cifmw_setup ------------------------------------------------------------- 1.40s
2026-03-14 09:31:22.522147 | controller | gather_facts ------------------------------------------------------------ 1.16s
2026-03-14 09:31:22.522152 | controller | discover_latest_image --------------------------------------------------- 1.15s
2026-03-14 09:31:22.522157 | controller | install_ca -------------------------------------------------------------- 0.43s
2026-03-14 09:31:22.522161 | controller | networking_mapper ------------------------------------------------------- 0.32s
2026-03-14 09:31:22.522165 | controller | ansible.builtin.set_fact ------------------------------------------------ 0.15s
2026-03-14 09:31:22.522170 | controller | ansible.builtin.include_role -------------------------------------------- 0.10s
2026-03-14 09:31:22.522176 | controller | ansible.builtin.assert -------------------------------------------------- 0.07s
2026-03-14 09:31:22.522182 | controller | ansible.builtin.include_vars -------------------------------------------- 0.07s
2026-03-14 09:31:22.522188 | controller | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2026-03-14 09:31:22.522194 | controller | total ----------------------------------------------------------------- 734.12s
2026-03-14 09:31:22.815613 | controller | changed
2026-03-14 09:31:22.882957 | 
2026-03-14 09:31:22.883091 | PLAY RECAP
2026-03-14 09:31:22.883144 | controller | ok: 1 changed: 1 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0
2026-03-14 09:31:22.883174 | 
2026-03-14 09:31:22.982226 | PRE-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/kuttl/kuttl-from-operator-deps.yaml@main]
2026-03-14 09:31:22.992513 | RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/kuttl/kuttl-from-operator-run.yaml@main]
2026-03-14 09:31:23.587903 | 
2026-03-14 09:31:23.588008 | PLAY [Run ci/playbooks/kuttl/run-kuttl-from-operator-targets.yaml]
2026-03-14 09:31:23.611468 | 
2026-03-14 09:31:23.611572 | TASK [Run kuttl tests from operator playbook]
2026-03-14 09:31:24.669886 | controller |
2026-03-14 09:31:24.702256 | controller | PLAY [controller] **************************************************************
2026-03-14 09:31:24.702318 | controller |
2026-03-14 09:31:24.702324 | controller | TASK [Run hooks before running kuttl tests name=run_hook] **********************
2026-03-14 09:31:24.702329 | controller | Saturday 14 March 2026  09:31:24 +0000 (0:00:00.046)       0:00:00.046 ********
2026-03-14 09:31:24.702333 | controller | Saturday 14 March 2026  09:31:24 +0000 (0:00:00.044)       0:00:00.044 ********
2026-03-14 09:31:24.702347 | controller |
2026-03-14 09:31:24.763344 | controller | TASK [run_hook : Assert parameters are valid quiet=True, that=['_list_hooks is not string', '_list_hooks is not mapping', '_list_hooks is iterable', '(hooks | default([])) is not string', '(hooks | default([])) is not mapping', '(hooks | default([])) is iterable']] ***
2026-03-14 09:31:24.763420 | controller | Saturday 14 March 2026  09:31:24 +0000 (0:00:00.032)       0:00:00.079 ********
2026-03-14 09:31:24.763426 | controller | Saturday 14 March 2026  09:31:24 +0000 (0:00:00.032)       0:00:00.077 ********
2026-03-14 09:31:24.763441 | controller | ok: [controller]
2026-03-14 09:31:24.858483 | controller |
2026-03-14 09:31:24.858568 | controller | TASK [run_hook : Assert single hooks are all mappings quiet=True, that=['_not_mapping_hooks | length == 0'], msg=All single hooks must be a list of mappings or a mapping.] ***
2026-03-14 09:31:24.858583 | controller | Saturday 14 March 2026  09:31:24 +0000 (0:00:00.061)       0:00:00.140 ********
2026-03-14 09:31:24.858588 | controller | Saturday 14 March 2026  09:31:24 +0000 (0:00:00.061)       0:00:00.138 ********
2026-03-14 09:31:24.858617 | controller | ok: [controller]
2026-03-14 09:31:24.990392 | controller |
2026-03-14 09:31:24.990454 | controller | TASK [run_hook : Loop on hooks for pre_kuttl_from_operator _raw_params={{ hook.type }}.yml] ***
2026-03-14 09:31:24.990460 | controller | Saturday 14 March 2026  09:31:24 +0000 (0:00:00.095)       0:00:00.235 ********
2026-03-14 09:31:24.990465 | controller | Saturday 14 March 2026  09:31:24 +0000 (0:00:00.095)       0:00:00.233 ********
2026-03-14 09:31:24.990480 | controller | skipping: [controller]
2026-03-14 09:56:26.488310 | controller |
2026-03-14 09:56:26.488391 | controller | TASK [run kuttl test suite from operator Makefile chdir={{ ansible_user_dir }}/{{ operator_basedir }}, _raw_params=make kuttl-test-run] ***
2026-03-14 09:56:26.488398 | controller | Saturday 14 March 2026  09:31:24 +0000 (0:00:00.131)       0:00:00.367 ********
2026-03-14 09:56:26.488404 | controller | Saturday 14 March 2026  09:31:24 +0000 (0:00:00.131)       0:00:00.365 ********
2026-03-14 09:56:26.488415 | controller | task path: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/ci/playbooks/kuttl/run-kuttl-from-operator-targets.yaml:16
2026-03-14 09:56:26.488455 | controller | fatal: [controller]: FAILED! =>
2026-03-14 09:56:26.488462 | controller |     changed: true
2026-03-14 09:56:26.488466 | controller |     cmd:
2026-03-14 09:56:26.488471 | controller |     - make
2026-03-14 09:56:26.488476 | controller |     - kuttl-test-run
2026-03-14 09:56:26.488480 | controller |     delta: '0:25:00.537852'
2026-03-14 09:56:26.488490 | controller |     end: '2026-03-14 09:56:25.827012'
2026-03-14 09:56:26.488496 | controller |     msg: non-zero return code
2026-03-14 09:56:26.488501 | controller |     rc: 2
2026-03-14 09:56:26.488505 | controller |     start: '2026-03-14 09:31:25.289160'
2026-03-14 09:56:26.488511 | controller |     stderr: |-
2026-03-14 09:56:26.488516 | controller |       Warning: spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`.
2026-03-14 09:56:26.488520 | controller |       Warning: The 'cinderBackup' field is deprecated and will be removed in a future release. Please migrate to 'cinderBackups'.
2026-03-14 09:56:26.488525 | controller |       make: *** [Makefile:444: kuttl-test-run] Error 1
2026-03-14 09:56:26.488529 | controller |     stderr_lines:
2026-03-14 09:56:26.488534 | controller |     - 'Warning: spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default
2026-03-14 09:56:26.488539 | controller |       value changed from `Never` to `Always`.'
2026-03-14 09:56:26.488543 | controller |     - 'Warning: The ''cinderBackup'' field is deprecated and will be removed in a future
2026-03-14 09:56:26.488548 | controller |       release. Please migrate to ''cinderBackups''.'
2026-03-14 09:56:26.488552 | controller |     - 'make: *** [Makefile:444: kuttl-test-run] Error 1'
2026-03-14 09:56:26.488566 | controller |     stdout: "oc kuttl test --v 1 --start-kind=false --config test/kuttl/test-suites/default/config.yaml\n===
2026-03-14 09:56:26.488571 | controller |       RUN   kuttl\n    harness.go:463: starting setup\n    harness.go:255: running tests
2026-03-14 09:56:26.488575 | controller |       using configured kubeconfig.\n    harness.go:278: Successful connection to cluster
2026-03-14 09:56:26.488580 | controller |       at: https://api.crc.testing:6443\n    harness.go:363: running tests\n    harness.go:75:
2026-03-14 09:56:26.488585 | controller |       going to run test suite with timeout of 300 seconds for each step\n    harness.go:375:
2026-03-14 09:56:26.488590 | controller |       testsuite: test/kuttl/test-suites/default/ has 11 tests\n=== RUN   kuttl/harness\n===
2026-03-14 09:56:26.488618 | controller |       RUN   kuttl/harness/appcred-tests\n=== PAUSE kuttl/harness/appcred-tests\n=== RUN
2026-03-14 09:56:26.488625 | controller |       \  kuttl/harness/common\n=== PAUSE kuttl/harness/common\n=== RUN   kuttl/harness/deps\n===
2026-03-14 09:56:26.488630 | controller |       PAUSE kuttl/harness/deps\n=== RUN   kuttl/harness/watcher\n=== PAUSE kuttl/harness/watcher\n===
2026-03-14 09:56:26.488635 | controller |       RUN   kuttl/harness/watcher-api-scaling\n=== PAUSE kuttl/harness/watcher-api-scaling\n===
2026-03-14 09:56:26.488639 | controller |       RUN   kuttl/harness/watcher-cinder\n=== PAUSE kuttl/harness/watcher-cinder\n===
2026-03-14 09:56:26.488644 | controller |       RUN   kuttl/harness/watcher-notification\n=== PAUSE kuttl/harness/watcher-notification\n===
2026-03-14 09:56:26.488648 | controller |       RUN   kuttl/harness/watcher-rmquser\n=== PAUSE kuttl/harness/watcher-rmquser\n===
2026-03-14 09:56:26.488653 | controller |       RUN   kuttl/harness/watcher-tls\n=== PAUSE kuttl/harness/watcher-tls\n=== RUN   kuttl/harness/watcher-tls-certs-change\n===
2026-03-14 09:56:26.488658 | controller |       PAUSE kuttl/harness/watcher-tls-certs-change\n=== RUN   kuttl/harness/watcher-topology\n===
2026-03-14 09:56:26.488664 | controller |       PAUSE kuttl/harness/watcher-topology\n=== CONT  kuttl/harness/appcred-tests\n    logger.go:42:
2026-03-14 09:56:26.488671 | controller |       09:31:25 | appcred-tests | Skipping creation of user-supplied namespace: watcher-kuttl-default\n
2026-03-14 09:56:26.488677 | controller |       \   logger.go:42: 09:31:25 | appcred-tests/0-cleanup | starting test step 0-cleanup\n
2026-03-14 09:56:26.488687 | controller |       \   logger.go:42: 09:31:25 | appcred-tests/0-cleanup | test step completed 0-cleanup\n
2026-03-14 09:56:26.488692 | controller |       \   logger.go:42: 09:31:25 | appcred-tests/1-deploy | starting test step 1-deploy\n
2026-03-14 09:56:26.488696 | controller |       \   logger.go:42: 09:31:25 | appcred-tests/1-deploy | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:56:26.488701 | controller |       created\n    logger.go:42: 09:31:25 | appcred-tests/1-deploy | running command:
2026-03-14 09:56:26.488705 | controller |       [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        oc wait -n \"${NS}\"
2026-03-14 09:56:26.488710 | controller |       watcher watcher-kuttl --for=condition=Ready --timeout=600s]\n    logger.go:42: 09:32:20
2026-03-14 09:56:26.488714 | controller |       | appcred-tests/1-deploy | watcher.watcher.openstack.org/watcher-kuttl condition
2026-03-14 09:56:26.488719 | controller |       met\n    logger.go:42: 09:32:20 | appcred-tests/1-deploy | test step completed 1-deploy\n
2026-03-14 09:56:26.488723 | controller |       \   logger.go:42: 09:32:20 | appcred-tests/2-deploy-appcred | starting test step
2026-03-14 09:56:26.488727 | controller |       2-deploy-appcred\n    logger.go:42: 09:32:20 | appcred-tests/2-deploy-appcred |
2026-03-14 09:56:26.488732 | controller |       running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.488736 | controller |       \       old_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.488741 | controller |       \       old_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.488745 | controller |       \       old_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.488750 | controller |       jsonpath='{.metadata.uid}')\n        oc create configmap appcred-watcher-pre \\\n
2026-03-14 09:56:26.488760 | controller |       \         --from-literal=api_uid=\"${old_api_uid}\" \\\n          --from-literal=applier_uid=\"${old_applier_uid}\"
2026-03-14 09:56:26.488765 | controller |       \\\n          --from-literal=decision_engine_uid=\"${old_de_uid}\" \\\n          --dry-run=client
2026-03-14 09:56:26.488770 | controller |       -o yaml | oc apply -n \"${NS}\" -f -\n        \n        cat <<'EOF' | oc apply -n
2026-03-14 09:56:26.488774 | controller |       \"${NS}\" -f -\n        apiVersion: keystone.openstack.org/v1beta1\n        kind:
2026-03-14 09:56:26.488779 | controller |       KeystoneApplicationCredential\n        metadata:\n          name: ac-watcher\n        spec:\n
2026-03-14 09:56:26.488783 | controller |       \         secret: osp-secret\n          passwordSelector: WatcherPassword\n          userName:
2026-03-14 09:56:26.488787 | controller |       watcher\n          roles:\n            - admin\n            - service\n          unrestricted:
2026-03-14 09:56:26.488792 | controller |       false\n        EOF\n        \n        oc patch watcher watcher-kuttl -n \"${NS}\"
2026-03-14 09:56:26.488796 | controller |       --type=merge -p '{\"spec\":{\"auth\":{\"applicationCredentialSecret\":\"ac-watcher-secret\"}}}']\n
2026-03-14 09:56:26.488801 | controller |       \   logger.go:42: 09:32:20 | appcred-tests/2-deploy-appcred | configmap/appcred-watcher-pre
2026-03-14 09:56:26.488805 | controller |       created\n    logger.go:42: 09:32:21 | appcred-tests/2-deploy-appcred | keystoneapplicationcredential.keystone.openstack.org/ac-watcher
2026-03-14 09:56:26.488810 | controller |       created\n    logger.go:42: 09:32:21 | appcred-tests/2-deploy-appcred | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:56:26.488814 | controller |       patched\n    logger.go:42: 09:32:21 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.488818 | controller |       [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n        echo \"Waiting
2026-03-14 09:56:26.488823 | controller |       for KeystoneApplicationCredential ac-watcher to be Ready...\"\n        oc wait -n
2026-03-14 09:56:26.488827 | controller |       \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n        \n        ac_id=$(oc
2026-03-14 09:56:26.488832 | controller |       get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n        if [
2026-03-14 09:56:26.488837 | controller |       -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.488843 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.488847 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.488852 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.488856 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.488861 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.488865 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.488869 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.488874 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.488878 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.488884 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.488889 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.488893 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.488898 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.488902 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.488909 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.488926 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.488930 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.488935 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.488939 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.488943 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.488948 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.488952 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.488956 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.488963 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.488968 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.488972 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.488977 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.488981 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.488986 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.488990 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.488994 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.488999 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.489003 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.489007 | controller |       09:37:21 | appcred-tests/2-deploy-appcred | error: timed out waiting for the condition
2026-03-14 09:56:26.489012 | controller |       on keystoneapplicationcredentials/ac-watcher\n    logger.go:42: 09:37:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.489016 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.489021 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.489025 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.489029 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.489034 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.489038 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.489042 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.489047 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.489051 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.489056 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.489060 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.489064 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.489073 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.489077 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.489082 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.489086 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.489090 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.489095 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.489099 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.489104 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.489108 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.489112 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.489117 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.489121 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.489126 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.489130 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.489134 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.489139 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.489143 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.489148 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.489154 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.489158 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.489163 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.489167 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.489172 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.489176 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.489180 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.489185 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.489189 | controller |       09:37:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.489193 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.489198 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.489202 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.489206 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.489213 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.489218 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.489222 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.489227 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.489231 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.489235 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.489240 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.489244 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.489248 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.489253 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.489257 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.489261 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.489266 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.489270 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.489275 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.489279 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.489283 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.489288 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.489292 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.489296 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.489301 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.489305 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.489309 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.489314 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.489318 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.489323 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.489330 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.489337 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.489344 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.489349 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.489355 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.489361 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.489365 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.489372 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.489376 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.489381 | controller |       09:37:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.489385 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.489389 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.489394 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.489398 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.489403 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.489407 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.489411 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.489416 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.489420 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.489424 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.489429 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.489433 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.489437 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.489442 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.489446 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.489450 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.489455 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.489459 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.489464 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.489478 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.489483 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.489487 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.489492 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.489496 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.489500 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.489505 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.489509 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.489513 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.489518 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.489525 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.489529 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.489533 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.489538 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.489545 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.489549 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.489553 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.489558 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.489562 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.489567 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.489571 | controller |       09:37:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.489575 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.489580 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.489584 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.489588 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.489618 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.489627 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.489631 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.489636 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.489640 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.489645 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.489649 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.489654 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.489661 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.489665 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.489670 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.489674 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.489679 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.489683 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.489688 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.489692 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.489699 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.489704 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.489708 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.489712 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.489717 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.489721 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.489727 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.489733 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.489739 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.489745 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.489751 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.489757 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.489763 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.489768 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.489775 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.489789 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.489794 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.489799 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.489805 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.489809 | controller |       09:37:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.489814 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.489818 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.489822 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.489827 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.489835 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.489840 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.489844 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.489849 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.489853 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.489857 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.489862 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.489866 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.489871 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.489877 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.489882 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.489886 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.489891 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.489895 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.489900 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.489904 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.489908 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.489913 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.489917 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.489922 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.489926 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.489930 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.489935 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.489939 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.489943 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.489948 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.489952 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.489956 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.489962 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.489967 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.489971 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.489976 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.489986 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.489991 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.489996 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.490000 | controller |       09:37:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.490005 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.490009 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.490013 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.490018 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.490024 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.490029 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.490033 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.490038 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.490042 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.490046 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.490051 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.490055 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.490059 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.490064 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.490068 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.490072 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.490077 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.490081 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.490086 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.490091 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.490096 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.490100 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.490104 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.490109 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.490113 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.490117 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.490122 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.490126 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.490130 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.490135 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.490139 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.490144 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.490148 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.490152 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.490157 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.490161 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.490165 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.490172 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.490183 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.490188 | controller |       09:37:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.490192 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.490197 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.490201 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.490205 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.490210 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.490214 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.490219 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.490223 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.490228 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.490232 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.490236 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.490242 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.490247 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.490251 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.490256 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.490262 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.490266 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.490271 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.490275 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.490280 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.490284 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.490288 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.490293 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.490297 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.490302 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.490306 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.490310 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.490315 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.490319 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.490326 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.490330 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.490335 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.490339 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.490343 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.490348 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.490352 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.490356 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.490361 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.490365 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.490376 | controller |       09:37:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.490380 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.490385 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.490389 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.490394 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.490398 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.490403 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.490407 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.490411 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.490416 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.490420 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.490424 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.490429 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.490433 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.490438 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.490442 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.490446 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.490451 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.490455 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.490459 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.490464 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.490470 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.490475 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.490479 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.490484 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.490488 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.490492 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.490497 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.490501 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.490508 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.490512 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.490517 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.490521 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.490525 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.490530 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.490534 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.490539 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.490543 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.490547 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.490552 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.490556 | controller |       09:37:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.490567 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.490571 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.490576 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.490580 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.490584 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.490589 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.490608 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.490615 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.490619 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.490624 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.490628 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.490633 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.490637 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.490644 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.490648 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.490653 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.490657 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.490662 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.490666 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.490670 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.490675 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.490679 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.490684 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.490688 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.490692 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.490697 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.490701 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.490706 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.490710 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.490714 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.490720 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.490725 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.490729 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.490734 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.490738 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.490743 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.490747 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.490751 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.490756 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.490760 | controller |       09:37:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.490765 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.490769 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.490781 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.490785 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.490792 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.490799 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.490804 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.490808 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.490812 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.490817 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.490821 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.490826 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.490830 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.490834 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.490839 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.490844 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.490850 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.490855 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.490859 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.490863 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.490868 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.490901 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.490906 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.490910 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.490914 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.490919 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.490923 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.490927 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.490932 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.490938 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.490943 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.490947 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.490951 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.490956 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.490960 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.490965 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.490969 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.490987 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.490991 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.490996 | controller |       09:37:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.491000 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.491005 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.491009 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.491019 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.491024 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.491028 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.491032 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.491037 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.491041 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.491046 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.491050 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.491055 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.491059 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.491063 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.491068 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.491072 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.491076 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.491081 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.491085 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.491090 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.491094 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.491098 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.491103 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.491107 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.491111 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.491116 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.491120 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.491126 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.491131 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.491137 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.491142 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.491146 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.491150 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.491155 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.491159 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.491164 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.491168 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.491173 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.491177 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.491181 | controller |       09:37:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.491186 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.491190 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.491194 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.491199 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.491210 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.491215 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.491219 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.491223 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.491228 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.491232 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.491237 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.491241 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.491245 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.491250 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.491254 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.491259 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.491263 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.491267 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.491272 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.491276 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.491282 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.491287 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.491291 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.491295 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.491300 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.491304 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.491308 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.491313 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.491317 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.491321 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.491326 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.491330 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.491335 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.491339 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.491343 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.491348 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.491352 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.491356 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.491361 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.491365 | controller |       09:37:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.491370 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.491374 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.491378 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.491383 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.491387 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.491391 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.491400 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.491405 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.491409 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.491415 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.491420 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.491424 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.491429 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.491435 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.491439 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.491444 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.491448 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.491452 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.491457 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.491461 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.491466 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.491470 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.491474 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.491479 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.491483 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.491488 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.491492 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.491496 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.491501 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.491505 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.491509 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.491514 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.491518 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.491522 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.491527 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.491531 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.491536 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.491540 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.491544 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.491549 | controller |       09:37:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.491553 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.491558 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.491562 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.491566 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.491572 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.491577 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.491581 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.491590 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.491607 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.491613 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.491618 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.491622 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.491627 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.491631 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.491636 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.491641 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.491646 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.491650 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.491655 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.491659 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.491664 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.491668 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.491673 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.491677 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.491681 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.491687 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.491691 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.491696 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.491700 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.491704 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.491709 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.491713 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.491717 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.491722 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.491726 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.491731 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.491735 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.491741 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.491746 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.491750 | controller |       09:37:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.491755 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.491759 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.491763 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.491768 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.491772 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.491776 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.491781 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.491785 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.491796 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.491800 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.491805 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.491809 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.491814 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.491818 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.491822 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.491827 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.491831 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.491836 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.491840 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.491844 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.491850 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.491854 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.491859 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.491863 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.491868 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.491872 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.491876 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.491881 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.491885 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.491891 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.491896 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.491900 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.491904 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.491909 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.491914 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.491920 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.491927 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.491932 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.491939 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.491944 | controller |       09:37:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.491948 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.491953 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.491957 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.491962 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.491966 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.491971 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.491975 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.491979 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.491985 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.491989 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.492000 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.492005 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.492009 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.492014 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.492018 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.492022 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.492027 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.492031 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.492036 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.492040 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.492046 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.492051 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.492055 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.492060 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.492064 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.492068 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.492073 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.492077 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.492082 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.492086 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.492092 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.492096 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.492101 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.492105 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.492109 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.492114 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.492118 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.492123 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.492127 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.492131 | controller |       09:37:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.492136 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.492140 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.492145 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.492149 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.492153 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.492158 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.492162 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.492167 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.492171 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.492175 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.492180 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.492189 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.492194 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.492200 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.492205 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.492209 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.492214 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.492218 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.492223 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.492227 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.492231 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.492236 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.492240 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.492245 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.492250 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.492254 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.492259 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.492263 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.492268 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.492272 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.492276 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.492281 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.492285 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.492290 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.492294 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.492298 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.492303 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.492307 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.492312 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.492316 | controller |       09:37:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.492320 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.492325 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.492330 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.492336 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.492342 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.492347 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.492353 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.492358 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.492362 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.492367 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.492371 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.492375 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.492380 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.492390 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.492395 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.492399 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.492403 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.492408 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.492412 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.492417 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.492421 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.492426 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.492430 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.492435 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.492439 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.492443 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.492448 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.492452 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.492456 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.492461 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.492465 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.492470 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.492474 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.492478 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.492483 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.492487 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.492492 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.492500 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.492504 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.492508 | controller |       09:37:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.492513 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.492517 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.492522 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.492527 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.492532 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.492537 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.492541 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.492546 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.492550 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.492554 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.492559 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.492563 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.492567 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.492572 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.492582 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.492586 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.492591 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.492609 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.492613 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.492627 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.492632 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.492636 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.492641 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.492645 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.492649 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.492654 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.492658 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.492662 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.492667 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.492674 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.492679 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.492683 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.492688 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.492692 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.492697 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.492701 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.492705 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.492710 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.492714 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.492719 | controller |       09:37:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.492723 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.492727 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.492732 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.492736 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.492741 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.492745 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.492749 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.492754 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.492758 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.492763 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.492767 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.492771 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.492776 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.492780 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.492784 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.492789 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.492800 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.492805 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.492809 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.492814 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.492822 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.492826 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.492831 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.492836 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.492841 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.492845 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.492849 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.492854 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.492858 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.492862 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.492867 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.492871 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.492876 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.492880 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.492884 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.492889 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.492893 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.492897 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.492902 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.492906 | controller |       09:37:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.492911 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.492915 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.492920 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.492924 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.492928 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.492933 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.492937 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.492941 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.492946 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.492950 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.492955 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.492959 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.492963 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.492970 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.492974 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.492979 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.492985 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.492998 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.493005 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.493010 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.493015 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.493019 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.493024 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.493028 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.493033 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.493037 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.493041 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.493046 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.493050 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.493055 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.493059 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.493063 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.493068 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.493072 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.493077 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.493081 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.493086 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.493090 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.493096 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.493100 | controller |       09:37:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.493105 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.493111 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.493117 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.493121 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.493128 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.493132 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.493136 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.493141 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.493145 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.493150 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.493154 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.493159 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.493163 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.493167 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.493172 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.493176 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.493180 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.493185 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.493195 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.493200 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.493204 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.493209 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.493213 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.493217 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.493222 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.493226 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.493231 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.493235 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.493239 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.493244 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.493248 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.493253 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.493257 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.493262 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.493266 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.493270 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.493275 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.493282 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.493286 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.493291 | controller |       09:37:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.493295 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.493300 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.493304 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.493308 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.493313 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.493317 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.493322 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.493326 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.493331 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.493335 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.493339 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.493344 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.493349 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.493354 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.493358 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.493362 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.493367 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.493371 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.493375 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.493386 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.493391 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.493395 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.493399 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.493404 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.493408 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.493413 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.493417 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.493421 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.493425 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.493433 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.493438 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.493443 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.493447 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.493451 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.493456 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.493460 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.493465 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.493469 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.493473 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.493478 | controller |       09:37:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.493482 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.493486 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.493491 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.493495 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.493499 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.493503 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.493508 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.493512 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.493516 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.493521 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.493525 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.493529 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.493533 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.493538 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.493542 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.493546 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.493551 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.493555 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.493559 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.493563 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.493569 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.493579 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.493584 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.493588 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.493593 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.493613 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.493618 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.493622 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.493627 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.493631 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.493635 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.493640 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.493644 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.493648 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.493653 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.493657 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.493661 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.493666 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.493670 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.493674 | controller |       09:37:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.493679 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.493684 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.493689 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.493693 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.493697 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.493702 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.493706 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.493710 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.493714 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.493719 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.493723 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.493727 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.493732 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.493739 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.493744 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.493748 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.493752 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.493757 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.493761 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.493765 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.493771 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.493775 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.493785 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.493790 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.493794 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.493798 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.493803 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.493807 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.493811 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.493815 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.493820 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.493824 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.493828 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.493833 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.493837 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.493841 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.493845 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.493850 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.493854 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.493858 | controller |       09:37:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.493862 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.493867 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.493871 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.493875 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.493882 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.493887 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.493891 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.493895 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.493900 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.493904 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.493908 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.493912 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.493916 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.493921 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.493925 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.493929 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.493934 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.493939 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.493943 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.493947 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.493951 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.493956 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.493960 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.493964 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.493974 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.493978 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.493983 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.493987 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.493991 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.493995 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.494000 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.494004 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.494008 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.494013 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.494017 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.494021 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.494025 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.494031 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.494036 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.494040 | controller |       09:37:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.494044 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.494049 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.494054 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.494060 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.494066 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.494072 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.494078 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.494083 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.494089 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.494095 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.494101 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.494105 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.494109 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.494114 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.494118 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.494122 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.494127 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.494131 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.494135 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.494139 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.494144 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.494148 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.494152 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.494156 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.494161 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.494171 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.494176 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.494180 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.494184 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.494191 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.494195 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.494200 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.494204 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.494208 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.494212 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.494217 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.494221 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.494225 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.494229 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.494233 | controller |       09:37:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.494239 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.494244 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.494248 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.494252 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.494256 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.494261 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.494265 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.494269 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.494274 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.494278 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.494282 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.494286 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.494291 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.494295 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.494299 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.494304 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.494309 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.494313 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.494317 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.494322 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.494327 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.494332 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.494337 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.494341 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.494345 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.494349 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.494354 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.494363 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.494368 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.494373 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.494378 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.494383 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.494388 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.494393 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.494399 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.494403 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.494407 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.494411 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.494416 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.494420 | controller |       09:37:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.494424 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.494428 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.494433 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.494437 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.494441 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.494446 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.494450 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.494454 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.494458 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.494463 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.494467 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.494471 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.494475 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.494481 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.494487 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.494491 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.494496 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.494500 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.494504 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.494508 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.494512 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.494517 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.494521 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.494525 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.494529 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.494534 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.494538 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.494542 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.494551 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.494556 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.494560 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.494565 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.494569 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.494573 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.494577 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.494582 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.494586 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.494590 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.494606 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.494612 | controller |       09:37:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.494617 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.494621 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.494625 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.494630 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.494638 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.494642 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.494646 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.494650 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.494655 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.494659 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.494663 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.494667 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.494671 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.494676 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.494680 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.494684 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.494688 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.494693 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.494697 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.494701 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.494707 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.494711 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.494715 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.494719 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.494724 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.494728 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.494732 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.494736 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.494740 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.494752 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.494757 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.494761 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.494765 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.494769 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.494774 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.494778 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.494782 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.494788 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.494794 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.494798 | controller |       09:37:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.494802 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.494807 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.494811 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.494815 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.494819 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.494824 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.494828 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.494832 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.494836 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.494841 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.494845 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.494849 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.494853 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.494858 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.494862 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.494866 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.494870 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.494888 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.494893 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.494897 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.494901 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.494905 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.494910 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.494914 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.494918 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.494922 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.494927 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.494931 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.494935 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.494941 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.494946 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.494955 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.494960 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.494964 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.494968 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.494973 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.494977 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.494981 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.494986 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.494990 | controller |       09:37:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.494994 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.494998 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.495003 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.495007 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.495011 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.495015 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.495020 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.495024 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.495028 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.495032 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.495036 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.495041 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.495045 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.495050 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.495055 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.495059 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.495063 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.495068 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.495072 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.495076 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.495082 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.495087 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.495091 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.495095 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.495099 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.495104 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.495108 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.495112 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.495116 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.495120 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.495125 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.495129 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.495139 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.495143 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.495149 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.495155 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.495161 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.495167 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.495172 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.495176 | controller |       09:37:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.495180 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:37:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.495185 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.495189 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.495193 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.495199 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.495204 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.495209 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.495213 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.495217 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.495221 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.495226 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.495230 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.495234 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.495241 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.495245 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.495249 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.495253 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.495258 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.495262 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.495266 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.495271 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.495275 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.495279 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.495283 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.495288 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.495292 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.495297 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.495301 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.495305 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.495309 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.495314 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.495318 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.495322 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.495332 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.495337 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.495342 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.495346 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.495351 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:37:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.495355 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.495359 | controller |       09:37:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.495363 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.495368 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.495372 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.495376 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.495382 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.495387 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.495392 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.495398 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.495402 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.495407 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.495412 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.495417 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.495422 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.495426 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.495430 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.495434 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.495438 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.495443 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.495447 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.495451 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.495455 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.495460 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.495464 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.495468 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.495472 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.495477 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.495481 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.495485 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.495489 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.495494 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.495498 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.495502 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.495506 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.495510 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.495515 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.495525 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.495529 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.495535 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.495540 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.495544 | controller |       09:38:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.495548 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.495552 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.495557 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.495561 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.495565 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.495569 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.495574 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.495578 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.495582 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.495586 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.495591 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.495606 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.495614 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.495618 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.495622 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.495628 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.495632 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.495636 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.495640 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.495645 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.495650 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.495654 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.495659 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.495663 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.495667 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.495671 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.495675 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.495680 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.495684 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.495690 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.495694 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.495699 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.495703 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.495707 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.495711 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.495716 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.495726 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.495730 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.495734 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.495739 | controller |       09:38:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.495743 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.495748 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.495752 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.495756 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.495760 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.495765 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.495769 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.495773 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.495778 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.495782 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.495786 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.495790 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.495795 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.495799 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.495803 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.495807 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.495812 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.495816 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.495820 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.495824 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.495830 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.495835 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.495839 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.495843 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.495848 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.495852 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.495856 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.495860 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.495865 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.495869 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.495873 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.495877 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.495883 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.495887 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.495892 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.495896 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.495900 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.495910 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.495914 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.495918 | controller |       09:38:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.495923 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.495927 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.495931 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.495936 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.495940 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.495944 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.495949 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.495953 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.495957 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.495961 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.495966 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.495970 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.495974 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.495982 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.495986 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.495990 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.495995 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.495999 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.496003 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.496008 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.496012 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.496016 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.496020 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.496025 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.496029 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.496033 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.496037 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.496041 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.496046 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.496050 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.496055 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.496059 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.496064 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.496068 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.496072 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.496076 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.496081 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.496085 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.496089 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.496099 | controller |       09:38:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.496103 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.496107 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.496112 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.496116 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.496122 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.496126 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.496131 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.496135 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.496139 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.496144 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.496148 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.496153 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.496158 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.496162 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.496166 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.496170 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.496174 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.496179 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.496183 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.496187 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.496192 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.496196 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.496200 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.496204 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.496209 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.496214 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.496220 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.496225 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.496231 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.496237 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.496243 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.496248 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.496254 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.496260 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.496264 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.496268 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.496273 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.496279 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.496283 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.496288 | controller |       09:38:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.496298 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.496303 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.496307 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.496312 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.496316 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.496320 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.496324 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.496329 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.496333 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.496337 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.496342 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.496346 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.496350 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.496354 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.496358 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.496363 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.496367 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.496371 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.496375 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.496380 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.496384 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.496388 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.496393 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.496397 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.496401 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.496405 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.496409 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.496414 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.496419 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.496426 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.496430 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.496434 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.496439 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.496443 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.496447 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.496451 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.496456 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.496460 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.496464 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.496468 | controller |       09:38:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.496473 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.496482 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.496487 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.496491 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.496508 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.496513 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.496518 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.496522 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.496526 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.496530 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.496535 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.496539 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.496543 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.496548 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.496552 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.496556 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.496560 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.496564 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.496569 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.496573 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.496582 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.496586 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.496591 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.496612 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.496619 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.496623 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.496627 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.496632 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.496636 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.496640 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.496644 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.496649 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.496653 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.496657 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.496661 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.496666 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.496670 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.496674 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.496678 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.496683 | controller |       09:38:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.496687 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.496691 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.496695 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.496706 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.496710 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.496715 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.496719 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.496723 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.496727 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.496732 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.496738 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.496742 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.496747 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.496753 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.496757 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.496761 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.496766 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.496770 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.496774 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.496778 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.496783 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.496787 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.496791 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.496795 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.496800 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.496804 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.496808 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.496812 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.496816 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.496821 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.496825 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.496829 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.496833 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.496838 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.496842 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.496846 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.496850 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.496855 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.496859 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.496863 | controller |       09:38:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.496867 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.496872 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.496876 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.496880 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.496891 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.496896 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.496900 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.496905 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.496909 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.496913 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.496917 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.496922 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.496926 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.496930 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.496934 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.496940 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.496945 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.496949 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.496954 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.496958 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.496962 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.496966 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.496970 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.496975 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.496979 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.496983 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.496987 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.496993 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.496998 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.497002 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.497006 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.497010 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.497015 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.497019 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.497023 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.497027 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.497031 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.497038 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.497042 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.497046 | controller |       09:38:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.497050 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.497055 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.497059 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.497063 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.497067 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.497072 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.497081 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.497086 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.497090 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.497094 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.497099 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.497103 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.497107 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.497112 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.497116 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.497120 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.497124 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.497128 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.497133 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.497137 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.497141 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.497145 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.497150 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.497154 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.497158 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.497162 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.497167 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.497171 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.497175 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.497181 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.497185 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.497190 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.497194 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.497198 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.497202 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.497207 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.497211 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.497215 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.497219 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.497224 | controller |       09:38:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.497228 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.497232 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.497236 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.497241 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.497245 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.497249 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.497253 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.497263 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.497267 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.497273 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.497278 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.497282 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.497286 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.497290 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.497295 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.497299 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.497304 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.497310 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.497316 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.497322 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.497330 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.497336 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.497342 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.497347 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.497352 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.497356 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.497360 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.497364 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.497368 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.497373 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.497379 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.497383 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.497388 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.497392 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.497396 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.497400 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.497405 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.497409 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.497415 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.497419 | controller |       09:38:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.497423 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.497428 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.497432 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.497436 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.497440 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.497445 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.497449 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.497453 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.497464 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.497469 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.497473 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.497477 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.497481 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.497488 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.497492 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.497496 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.497500 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.497505 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.497509 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.497513 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.497519 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.497523 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.497528 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.497532 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.497536 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.497543 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.497548 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.497552 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.497556 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.497560 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.497565 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.497569 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.497573 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.497578 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.497582 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.497586 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.497590 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.497606 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.497613 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.497617 | controller |       09:38:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.497621 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.497625 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.497630 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.497634 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.497640 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.497645 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.497649 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.497653 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.497657 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.497662 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.497672 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.497677 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.497681 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.497685 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.497690 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.497694 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.497698 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.497702 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.497707 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.497711 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.497715 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.497719 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.497724 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.497728 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.497732 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.497736 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.497741 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.497745 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.497749 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.497753 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.497758 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.497762 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.497766 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.497770 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.497775 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.497779 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.497783 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.497789 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.497794 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.497798 | controller |       09:38:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.497802 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.497806 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.497811 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.497815 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.497821 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.497825 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.497830 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.497834 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.497839 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.497844 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.497848 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.497859 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.497863 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.497867 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.497872 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.497876 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.497880 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.497884 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.497889 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.497893 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.497897 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.497901 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.497906 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.497910 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.497914 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.497918 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.497922 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.497927 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.497931 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.497937 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.497941 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.497945 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.497950 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.497954 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.497958 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.497962 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.497966 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.497971 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.497975 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.497979 | controller |       09:38:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.497983 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.497988 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.497992 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.497996 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.498000 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.498007 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.498011 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.498015 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.498020 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.498024 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.498028 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.498032 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.498036 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.498046 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.498050 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.498055 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.498059 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.498063 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.498067 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.498072 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.498078 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.498082 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.498086 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.498090 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.498096 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.498100 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.498105 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.498109 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.498113 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.498117 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.498122 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.498126 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.498130 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.498134 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.498139 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.498143 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.498147 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.498151 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.498156 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.498160 | controller |       09:38:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.498164 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.498168 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.498172 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.498177 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.498181 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.498185 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.498189 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.498194 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.498198 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.498202 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.498206 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.498210 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.498215 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.498223 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.498233 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.498239 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.498243 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.498248 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.498252 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.498256 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.498261 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.498265 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.498269 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.498273 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.498278 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.498282 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.498286 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.498290 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.498295 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.498299 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.498303 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.498307 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.498311 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.498316 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.498320 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.498324 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.498329 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.498333 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.498337 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.498341 | controller |       09:38:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.498346 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.498350 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.498354 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.498360 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.498366 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.498371 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.498375 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.498379 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.498383 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.498388 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.498392 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.498396 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.498400 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.498404 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.498409 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.498413 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.498424 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.498431 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.498437 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.498443 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.498452 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.498457 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.498461 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.498466 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.498470 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.498474 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.498478 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.498483 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.498487 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.498491 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.498495 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.498500 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.498504 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.498508 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.498512 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.498516 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.498521 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.498527 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.498531 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.498536 | controller |       09:38:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.498540 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.498544 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.498548 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.498553 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.498557 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.498561 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.498565 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.498570 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.498574 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.498578 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.498582 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.498586 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.498591 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.498606 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.498613 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.498617 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.498621 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.498632 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.498637 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.498641 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.498645 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.498649 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.498654 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.498662 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.498667 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.498671 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.498675 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.498679 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.498684 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.498690 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.498696 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.498700 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.498704 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.498708 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.498713 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.498717 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.498721 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.498725 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.498729 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.498734 | controller |       09:38:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.498738 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.498742 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.498746 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.498751 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.498755 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.498759 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.498763 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.498768 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.498772 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.498776 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.498780 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.498784 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.498789 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.498793 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.498797 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.498801 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.498805 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.498810 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.498820 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.498824 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.498830 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.498835 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.498839 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.498843 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.498847 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.498852 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.498856 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.498860 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.498864 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.498868 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.498873 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.498891 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.498897 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.498901 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.498905 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.498910 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.498914 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.498918 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.498922 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.498927 | controller |       09:38:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.498931 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.498935 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.498942 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.498946 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.498950 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.498954 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.498959 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.498963 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.498967 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.498971 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.498976 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.498980 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.498984 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.498990 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.498995 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.498999 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.499003 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.499007 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.499012 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.499022 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.499026 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.499031 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.499035 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.499039 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.499043 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.499048 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.499052 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.499056 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.499060 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.499065 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.499069 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.499073 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.499078 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.499082 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.499086 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.499090 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.499095 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.499099 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.499103 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.499107 | controller |       09:38:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.499112 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.499116 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.499120 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.499124 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.499132 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.499136 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.499141 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.499145 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.499149 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.499153 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.499158 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.499162 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.499166 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.499170 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.499175 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.499179 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.499183 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.499187 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.499192 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.499196 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.499207 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.499212 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.499216 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.499221 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.499225 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.499229 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.499234 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.499238 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.499242 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.499246 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.499251 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.499255 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.499259 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.499263 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.499267 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.499272 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.499276 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.499282 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.499286 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.499290 | controller |       09:38:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.499295 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.499299 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.499303 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.499307 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.499312 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.499316 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.499320 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.499324 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.499329 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.499333 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.499337 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.499341 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.499346 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.499350 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.499354 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.499358 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.499362 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.499367 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.499371 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.499375 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.499381 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.499386 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.499395 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.499400 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.499404 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.499408 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.499412 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.499416 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.499421 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.499427 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.499432 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.499436 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.499440 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.499444 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.499449 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.499453 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.499457 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.499461 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.499466 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.499470 | controller |       09:38:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.499474 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.499480 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.499484 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.499489 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.499493 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.499498 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.499503 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.499507 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.499511 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.499516 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.499520 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.499524 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.499528 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.499532 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.499537 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.499541 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.499547 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.499551 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.499555 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.499560 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.499567 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.499571 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.499575 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.499580 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.499589 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.499604 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.499612 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.499616 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.499621 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.499625 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.499629 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.499634 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.499638 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.499642 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.499646 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.499651 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.499655 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.499659 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.499664 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.499668 | controller |       09:38:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.499672 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.499676 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.499681 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.499685 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.499689 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.499693 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.499698 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.499702 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.499706 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.499710 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.499715 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.499719 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.499723 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.499730 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.499734 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.499738 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.499745 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.499749 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.499753 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.499758 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.499762 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.499766 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.499771 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.499775 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.499779 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.499789 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.499794 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.499798 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.499802 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.499807 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.499811 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.499815 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.499820 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.499824 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.499828 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.499832 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.499837 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.499841 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.499845 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.499849 | controller |       09:38:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.499854 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.499858 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.499862 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.499866 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.499873 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.499877 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.499881 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.499885 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.499890 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.499894 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.499898 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.499902 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.499907 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.499911 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.499915 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.499919 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.499924 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.499928 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.499932 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.499936 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.499940 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.499945 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.499949 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.499953 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.499957 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.499962 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.499971 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.499975 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.499980 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.499984 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.499990 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.499994 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.499998 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.500003 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.500007 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.500011 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.500015 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.500021 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.500026 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.500030 | controller |       09:38:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.500036 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.500040 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.500044 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.500049 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.500053 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.500057 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.500061 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.500066 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.500070 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.500074 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.500078 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.500082 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.500086 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.500091 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.500095 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.500099 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.500103 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.500108 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.500112 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.500116 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.500120 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.500124 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.500129 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.500133 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.500137 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.500141 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.500146 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.500150 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.500159 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.500166 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.500170 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.500174 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.500179 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.500183 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.500187 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.500191 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.500196 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.500200 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.500204 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.500208 | controller |       09:38:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.500213 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.500217 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.500221 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.500225 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.500230 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.500234 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.500238 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.500242 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.500247 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.500251 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.500255 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.500259 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.500264 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.500268 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.500274 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.500279 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.500283 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.500287 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.500291 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.500296 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.500303 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.500308 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.500312 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.500316 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.500320 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.500325 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.500329 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.500333 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.500337 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.500347 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.500351 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.500355 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.500360 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.500364 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.500368 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.500373 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.500377 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.500381 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.500385 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.500390 | controller |       09:38:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.500394 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.500398 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.500402 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.500407 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.500412 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.500417 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.500421 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.500425 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.500430 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.500434 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.500438 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.500442 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.500447 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.500453 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.500457 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.500461 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.500465 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.500469 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.500474 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.500478 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.500482 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.500487 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.500491 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.500495 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.500499 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.500504 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.500508 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.500512 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.500516 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.500522 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.500527 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.500536 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.500541 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.500545 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.500550 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.500554 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.500558 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.500562 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.500568 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.500573 | controller |       09:38:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.500577 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.500582 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.500589 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.500618 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.500630 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.500635 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.500639 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.500643 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.500648 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.500652 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.500656 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.500660 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.500665 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.500669 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.500673 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.500677 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.500681 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.500686 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.500690 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.500694 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.500698 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.500703 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.500707 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.500711 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.500715 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.500719 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.500724 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.500728 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.500732 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.500736 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.500740 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.500745 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.500756 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.500760 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.500764 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.500769 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.500773 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.500779 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.500783 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.500788 | controller |       09:38:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.500792 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.500796 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.500800 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.500805 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.500809 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.500813 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.500817 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.500822 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.500826 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.500830 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.500834 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.500838 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.500843 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.500849 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.500853 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.500859 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.500863 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.500868 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.500872 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.500876 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.500880 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.500885 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.500889 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.500893 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.500897 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.500901 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.500906 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.500910 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.500914 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.500920 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.500924 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.500928 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.500933 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.500942 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.500947 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.500951 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.500955 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.500960 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.500964 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.500968 | controller |       09:38:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.500972 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.500977 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.500981 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.500985 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.500989 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.500993 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.500998 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.501002 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.501006 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.501010 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.501014 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.501019 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.501023 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.501027 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.501031 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.501035 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.501040 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.501044 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.501048 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.501052 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.501058 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.501063 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.501067 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.501071 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.501075 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.501079 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.501084 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.501088 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.501092 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.501096 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.501100 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.501105 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.501109 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.501113 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.501124 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.501129 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.501133 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.501137 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.501142 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.501146 | controller |       09:38:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.501150 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.501155 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.501159 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.501163 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.501167 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.501172 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.501176 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.501180 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.501184 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.501189 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.501193 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.501197 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.501201 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.501208 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.501212 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.501216 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.501220 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.501225 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.501229 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.501233 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.501239 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.501244 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.501248 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.501252 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.501257 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.501261 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.501265 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.501269 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.501273 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.501278 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.501284 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.501288 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.501292 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.501296 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.501301 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.501305 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.501314 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.501319 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.501323 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.501328 | controller |       09:38:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.501332 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.501336 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.501341 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.501345 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.501351 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.501355 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.501359 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.501364 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.501368 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.501372 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.501376 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.501381 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.501386 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.501391 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.501395 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.501399 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.501403 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.501408 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.501412 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.501416 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.501420 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.501425 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.501429 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.501433 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.501437 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.501442 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.501446 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.501450 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.501454 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.501458 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.501463 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.501467 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.501471 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.501475 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.501480 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.501484 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.501488 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.501499 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.501504 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.501508 | controller |       09:38:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.501513 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.501517 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.501521 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.501525 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.501530 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.501534 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.501538 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.501542 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.501547 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.501551 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.501555 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.501559 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.501564 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.501568 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.501572 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.501576 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.501580 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.501585 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.501589 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.501605 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.501611 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.501616 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.501620 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.501624 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.501628 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.501632 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.501637 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.501641 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.501645 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.501651 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.501656 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.501660 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.501666 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.501670 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.501675 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.501679 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.501685 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.501691 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.501701 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.501715 | controller |       09:38:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.501721 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.501725 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.501730 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.501734 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.501740 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.501744 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.501749 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.501753 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.501757 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.501761 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.501766 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.501770 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.501774 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.501778 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.501782 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.501787 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.501791 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.501795 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.501799 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.501804 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.501810 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.501814 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.501818 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.501823 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.501827 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.501831 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.501835 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.501840 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.501844 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.501848 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.501852 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.501857 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.501861 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.501865 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.501869 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.501874 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.501878 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.501882 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.501886 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.501891 | controller |       09:38:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.501900 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.501905 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.501909 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.501913 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.501918 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.501922 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.501926 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.501930 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.501935 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.501939 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.501943 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.501949 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.501953 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.501960 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.501964 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.501968 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.501972 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.501977 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.501981 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.501985 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.501989 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.501994 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.501998 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.502002 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.502006 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.502011 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.502015 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.502019 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.502023 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.502027 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.502032 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.502036 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.502040 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.502046 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.502051 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.502055 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.502059 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.502063 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.502067 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.502072 | controller |       09:38:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.502076 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.502086 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.502090 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.502094 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.502101 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.502105 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.502109 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.502113 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.502117 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.502122 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.502126 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.502130 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.502134 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.502139 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.502143 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.502149 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.502153 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.502157 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.502161 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.502166 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.502171 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.502176 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.502180 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.502184 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.502188 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.502193 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.502197 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.502201 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.502207 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.502211 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.502215 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.502219 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.502224 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.502228 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.502232 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.502236 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.502241 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.502247 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.502251 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.502255 | controller |       09:38:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.502259 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.502264 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.502268 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.502278 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.502282 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.502286 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.502291 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.502295 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.502299 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.502303 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.502308 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.502312 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.502316 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.502320 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.502325 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.502329 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.502333 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.502337 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.502342 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.502346 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.502350 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.502354 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.502359 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.502363 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.502367 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.502371 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.502376 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.502380 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.502384 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.502390 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.502394 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.502399 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.502403 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.502407 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.502411 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.502415 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.502420 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.502424 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.502428 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.502432 | controller |       09:38:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.502437 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.502441 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.502445 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.502449 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.502458 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.502463 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.502467 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.502471 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.502476 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.502480 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.502486 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.502490 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.502495 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.502499 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.502503 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.502507 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.502511 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.502516 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.502520 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.502524 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.502530 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.502534 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.502539 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.502543 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.502547 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.502551 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.502556 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.502560 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.502564 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.502568 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.502574 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.502578 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.502583 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.502587 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.502591 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.502609 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.502614 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.502618 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.502622 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.502626 | controller |       09:38:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.502631 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.502635 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.502639 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.502643 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.502648 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.502658 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.502662 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.502667 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.502671 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.502675 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.502679 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.502684 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.502688 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.502695 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.502700 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.502704 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.502708 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.502712 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.502717 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.502721 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.502725 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.502729 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.502734 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.502738 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.502742 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.502746 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.502750 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.502757 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.502761 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.502767 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.502773 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.502779 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.502784 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.502789 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.502793 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.502797 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.502801 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.502806 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.502810 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.502814 | controller |       09:38:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.502818 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.502823 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.502827 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.502831 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.502838 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.502842 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.502846 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.502856 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.502861 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.502865 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.502870 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.502874 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.502892 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.502897 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.502901 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.502905 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.502909 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.502914 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.502918 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.502922 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.502926 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.502931 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.502935 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.502939 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.502943 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.502947 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.502952 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.502956 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.502960 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.502964 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.502968 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.502973 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.502977 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.502981 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.502985 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.502990 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.502994 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.503000 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.503005 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.503009 | controller |       09:38:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.503013 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.503017 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.503022 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.503026 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.503032 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.503037 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.503041 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.503045 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.503055 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.503061 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.503065 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.503070 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.503074 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.503078 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.503082 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.503086 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.503091 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.503095 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.503099 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.503103 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.503110 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.503114 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.503118 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.503122 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.503127 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.503131 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.503135 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.503139 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.503144 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.503150 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.503154 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.503159 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.503163 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.503167 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.503171 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.503176 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.503180 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.503184 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.503188 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.503193 | controller |       09:38:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.503197 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.503201 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.503205 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.503210 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.503214 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.503218 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.503222 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.503227 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.503231 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.503235 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.503245 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.503249 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.503253 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.503258 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.503262 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.503266 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.503271 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.503275 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.503279 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.503283 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.503289 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.503294 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.503298 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.503302 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.503306 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.503312 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.503317 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.503321 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.503325 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.503329 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.503334 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.503338 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.503342 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.503347 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.503351 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.503355 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.503359 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.503363 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.503368 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.503372 | controller |       09:38:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.503376 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.503380 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.503385 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.503389 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.503393 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.503397 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.503402 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.503406 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.503410 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.503414 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.503418 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.503428 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.503432 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.503438 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.503443 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.503449 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.503453 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.503457 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.503462 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.503466 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.503470 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.503474 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.503478 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.503483 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.503487 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.503491 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.503495 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.503499 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.503504 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.503508 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.503512 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.503516 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.503521 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.503525 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.503529 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.503533 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.503538 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.503542 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.503546 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.503550 | controller |       09:38:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.503555 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.503559 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.503563 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.503567 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.503573 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.503578 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.503582 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.503586 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.503592 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.503610 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.503614 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.503619 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.503623 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.503633 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.503638 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.503642 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.503646 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.503651 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.503655 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.503659 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.503663 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.503668 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.503672 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.503676 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.503680 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.503685 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.503689 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.503693 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.503697 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.503701 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.503706 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.503710 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.503714 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.503718 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.503723 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.503727 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.503731 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.503741 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.503745 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.503750 | controller |       09:38:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.503754 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.503758 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.503762 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.503767 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.503771 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.503777 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.503781 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.503785 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.503790 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.503794 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.503798 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.503802 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.503806 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.503811 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.503820 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.503825 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.503830 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.503836 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.503842 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.503847 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.503853 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.503859 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.503864 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.503870 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.503876 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.503881 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.503886 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.503892 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.503897 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.503903 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.503907 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.503912 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.503917 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.503922 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.503926 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.503930 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.503934 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.503938 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.503943 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.503947 | controller |       09:38:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.503951 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.503955 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.503960 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.503964 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.503968 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.503972 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.503977 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.503981 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.503985 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.503989 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.503994 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.503998 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.504002 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.504006 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.504011 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.504022 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.504026 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.504031 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.504035 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.504040 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.504047 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.504052 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.504056 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.504060 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.504064 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.504068 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.504073 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.504077 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.504081 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.504085 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.504090 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.504094 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.504098 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.504102 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.504107 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.504111 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.504115 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.504119 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.504124 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.504128 | controller |       09:38:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.504132 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.504137 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.504141 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.504147 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.504151 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.504155 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.504159 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.504164 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.504168 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.504172 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.504176 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.504181 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.504185 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.504191 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.504195 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.504199 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.504203 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.504213 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.504218 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.504223 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.504227 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.504232 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.504236 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.504240 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.504244 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.504249 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.504253 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.504257 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.504261 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.504266 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.504270 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.504274 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.504278 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.504283 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.504287 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.504291 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.504295 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.504300 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.504304 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.504308 | controller |       09:38:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.504312 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.504317 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.504321 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.504325 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.504333 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.504337 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.504341 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.504346 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.504350 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.504354 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.504358 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.504363 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.504367 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.504371 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.504375 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.504380 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.504384 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.504388 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.504398 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.504402 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.504406 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.504411 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.504415 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.504421 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.504425 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.504429 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.504434 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.504438 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.504442 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.504446 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.504451 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.504455 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.504459 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.504463 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.504467 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.504472 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.504476 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.504482 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.504486 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.504490 | controller |       09:38:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.504495 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.504499 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.504503 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.504507 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.504512 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.504516 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.504520 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.504524 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.504529 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.504533 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.504537 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.504541 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.504546 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.504550 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.504554 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.504558 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.504562 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.504567 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.504571 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.504580 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.504585 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.504589 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.504604 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.504611 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.504616 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.504620 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.504624 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.504628 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.504633 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.504639 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.504644 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.504648 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.504652 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.504656 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.504661 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.504665 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.504669 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.504673 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.504678 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.504682 | controller |       09:38:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.504686 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:38:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.504690 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.504696 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.504701 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.504705 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.504709 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.504713 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.504718 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.504722 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.504726 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.504731 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.504735 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.504739 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.504743 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.504747 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.504754 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.504759 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.504763 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.504767 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.504772 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.504784 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.504788 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.504793 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.504797 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.504802 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.504806 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.504810 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.504814 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.504819 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.504823 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.504827 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.504831 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.504835 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.504840 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.504844 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.504848 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.504853 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.504857 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:38:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.504861 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.504865 | controller |       09:38:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.504870 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.504874 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.504878 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.504882 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.504887 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.504891 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.504895 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.504899 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.504904 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.504908 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.504912 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.504916 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.504921 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.504929 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.504935 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.504941 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.504946 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.504950 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.504955 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.504959 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.504965 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.504975 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.504985 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.504990 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.504994 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.504999 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.505003 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.505007 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.505012 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.505016 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.505020 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.505024 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.505028 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.505033 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.505037 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.505041 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.505045 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.505050 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.505054 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.505058 | controller |       09:39:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.505062 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.505067 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.505071 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.505075 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.505081 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.505086 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.505090 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.505094 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.505098 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.505103 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.505107 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.505111 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.505115 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.505120 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.505124 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.505128 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.505132 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.505137 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.505141 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.505145 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.505149 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.505154 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.505158 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.505167 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.505172 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.505176 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.505180 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.505185 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.505189 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.505193 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.505199 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.505204 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.505208 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.505212 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.505216 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.505221 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.505225 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.505231 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.505235 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.505239 | controller |       09:39:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.505244 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.505249 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.505254 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.505258 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.505262 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.505267 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.505271 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.505275 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.505280 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.505284 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.505288 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.505292 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.505296 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.505301 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.505305 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.505309 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.505313 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.505318 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.505322 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.505326 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.505330 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.505335 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.505339 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.505343 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.505347 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.505358 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.505362 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.505367 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.505371 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.505377 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.505381 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.505386 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.505390 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.505394 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.505399 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.505403 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.505407 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.505411 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.505415 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.505420 | controller |       09:39:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.505424 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.505428 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.505433 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.505437 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.505441 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.505445 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.505450 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.505454 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.505458 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.505463 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.505467 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.505471 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.505475 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.505479 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.505484 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.505488 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.505494 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.505498 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.505503 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.505507 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.505513 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.505517 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.505521 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.505526 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.505530 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.505534 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.505544 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.505548 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.505552 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.505557 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.505561 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.505565 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.505569 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.505574 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.505578 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.505582 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.505586 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.505591 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.505606 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.505612 | controller |       09:39:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.505617 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.505621 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.505625 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.505629 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.505636 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.505640 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.505644 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.505648 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.505653 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.505657 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.505661 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.505665 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.505670 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.505676 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.505680 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.505684 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.505689 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.505693 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.505697 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.505701 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.505706 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.505710 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.505714 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.505718 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.505723 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.505727 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.505731 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.505735 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.505745 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.505750 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.505754 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.505758 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.505763 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.505767 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.505771 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.505775 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.505780 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.505784 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.505788 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.505792 | controller |       09:39:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.505798 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.505803 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.505807 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.505811 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.505818 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.505823 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.505827 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.505831 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.505836 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.505840 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.505844 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.505848 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.505853 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.505857 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.505861 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.505865 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.505870 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.505874 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.505878 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.505882 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.505888 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.505893 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.505897 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.505901 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.505905 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.505910 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.505914 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.505918 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.505922 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.505932 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.505936 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.505940 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.505945 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.505949 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.505954 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.505958 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.505962 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.505968 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.505974 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.505978 | controller |       09:39:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.505983 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.505987 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.505993 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.505999 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.506005 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.506011 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.506016 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.506022 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.506028 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.506033 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.506037 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.506041 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.506046 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.506050 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.506056 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.506062 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.506066 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.506070 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.506075 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.506079 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.506083 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.506087 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.506092 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.506096 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.506100 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.506105 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.506109 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.506113 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.506117 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.506124 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.506134 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.506138 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.506143 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.506147 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.506151 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.506156 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.506160 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.506164 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.506168 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.506173 | controller |       09:39:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.506177 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.506181 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.506185 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.506189 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.506194 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.506198 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.506202 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.506206 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.506211 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.506215 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.506219 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.506223 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.506227 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.506232 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.506236 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.506240 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.506244 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.506249 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.506253 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.506257 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.506263 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.506267 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.506272 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.506276 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.506280 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.506284 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.506289 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.506293 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.506297 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.506301 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.506306 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.506310 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.506319 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.506324 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.506328 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.506332 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.506337 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.506341 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.506347 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.506351 | controller |       09:39:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.506356 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.506360 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.506364 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.506368 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.506373 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.506377 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.506381 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.506385 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.506389 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.506394 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.506398 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.506402 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.506406 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.506413 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.506417 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.506421 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.506425 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.506430 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.506434 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.506438 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.506442 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.506447 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.506451 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.506455 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.506459 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.506464 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.506468 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.506472 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.506476 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.506480 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.506487 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.506492 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.506496 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.506506 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.506511 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.506515 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.506519 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.506524 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.506528 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.506532 | controller |       09:39:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.506537 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.506541 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.506545 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.506549 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.506555 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.506559 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.506564 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.506568 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.506572 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.506576 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.506581 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.506585 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.506589 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.506606 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.506613 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.506617 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.506622 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.506626 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.506630 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.506634 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.506639 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.506643 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.506647 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.506651 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.506656 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.506660 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.506664 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.506668 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.506673 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.506677 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.506681 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.506685 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.506689 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.506694 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.506704 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.506708 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.506713 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.506719 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.506723 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.506728 | controller |       09:39:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.506732 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.506736 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.506740 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.506745 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.506749 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.506753 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.506757 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.506762 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.506766 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.506770 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.506774 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.506779 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.506783 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.506787 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.506791 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.506796 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.506800 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.506804 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.506808 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.506812 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.506818 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.506823 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.506827 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.506831 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.506836 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.506840 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.506844 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.506848 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.506852 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.506860 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.506864 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.506868 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.506873 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.506877 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.506893 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.506899 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.506910 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.506914 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.506918 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.506923 | controller |       09:39:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.506927 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.506931 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.506936 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.506940 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.506946 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.506950 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.506955 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.506959 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.506963 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.506967 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.506971 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.506976 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.506980 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.506984 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.506988 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.506992 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.506997 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.507001 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.507005 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.507010 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.507016 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.507020 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.507024 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.507028 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.507033 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.507037 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.507041 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.507045 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.507050 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.507054 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.507058 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.507062 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.507067 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.507071 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.507075 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.507079 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.507084 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.507093 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.507099 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.507105 | controller |       09:39:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.507111 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.507116 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.507122 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.507126 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.507130 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.507135 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.507139 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.507143 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.507147 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.507151 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.507156 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.507160 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.507166 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.507173 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.507177 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.507181 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.507185 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.507190 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.507194 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.507198 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.507202 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.507207 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.507211 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.507215 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.507219 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.507224 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.507228 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.507232 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.507236 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.507240 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.507245 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.507249 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.507253 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.507258 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.507262 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.507266 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.507270 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.507275 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.507285 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.507290 | controller |       09:39:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.507294 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.507298 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.507303 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.507307 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.507313 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.507317 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.507322 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.507326 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.507330 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.507334 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.507338 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.507343 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.507347 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.507351 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.507355 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.507361 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.507366 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.507370 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.507374 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.507378 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.507383 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.507387 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.507391 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.507395 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.507400 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.507404 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.507408 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.507412 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.507417 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.507421 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.507425 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.507429 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.507435 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.507439 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.507444 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.507448 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.507452 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.507458 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.507462 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.507467 | controller |       09:39:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.507477 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.507482 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.507486 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.507491 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.507495 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.507499 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.507503 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.507508 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.507512 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.507516 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.507520 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.507525 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.507529 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.507533 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.507537 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.507541 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.507546 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.507550 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.507554 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.507558 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.507563 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.507567 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.507571 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.507575 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.507579 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.507584 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.507588 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.507592 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.507610 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.507616 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.507620 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.507625 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.507629 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.507633 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.507637 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.507642 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.507646 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.507650 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.507654 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.507659 | controller |       09:39:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.507663 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.507672 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.507677 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.507681 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.507686 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.507690 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.507694 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.507698 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.507703 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.507707 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.507711 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.507717 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.507722 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.507726 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.507730 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.507734 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.507739 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.507743 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.507747 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.507751 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.507762 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.507767 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.507771 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.507775 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.507779 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.507784 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.507788 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.507792 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.507796 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.507801 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.507807 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.507811 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.507815 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.507822 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.507827 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.507831 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.507835 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.507840 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.507844 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.507848 | controller |       09:39:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.507852 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.507857 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.507867 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.507872 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.507876 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.507880 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.507885 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.507889 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.507893 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.507897 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.507901 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.507906 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.507910 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.507916 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.507920 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.507925 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.507929 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.507933 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.507937 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.507942 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.507946 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.507950 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.507954 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.507959 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.507963 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.507967 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.507971 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.507976 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.507981 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.507986 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.507990 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.507994 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.507998 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.508003 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.508007 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.508011 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.508015 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.508020 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.508024 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.508028 | controller |       09:39:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.508032 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.508037 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.508041 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.508045 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.508057 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.508063 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.508068 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.508072 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.508076 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.508081 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.508085 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.508089 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.508093 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.508098 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.508102 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.508106 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.508110 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.508115 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.508119 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.508123 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.508127 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.508132 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.508136 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.508140 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.508144 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.508149 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.508153 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.508157 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.508161 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.508167 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.508173 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.508179 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.508185 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.508190 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.508194 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.508198 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.508203 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.508209 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.508214 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.508218 | controller |       09:39:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.508222 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.508226 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.508231 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.508235 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.508241 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.508251 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.508256 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.508260 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.508265 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.508269 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.508275 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.508279 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.508283 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.508288 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.508292 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.508296 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.508300 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.508305 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.508309 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.508313 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.508317 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.508321 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.508326 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.508330 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.508334 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.508338 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.508343 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.508347 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.508351 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.508357 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.508362 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.508366 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.508370 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.508374 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.508379 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.508383 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.508387 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.508391 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.508395 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.508400 | controller |       09:39:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.508404 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.508408 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.508412 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.508417 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.508421 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.508425 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.508429 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.508439 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.508443 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.508447 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.508451 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.508456 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.508460 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.508464 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.508468 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.508473 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.508477 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.508481 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.508485 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.508490 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.508496 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.508500 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.508504 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.508508 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.508513 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.508517 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.508521 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.508528 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.508533 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.508537 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.508541 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.508545 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.508550 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.508554 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.508558 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.508562 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.508567 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.508571 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.508575 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.508579 | controller |       09:39:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.508584 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.508588 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.508592 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.508610 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.508614 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.508618 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.508623 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.508627 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.508637 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.508641 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.508646 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.508650 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.508654 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.508660 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.508665 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.508672 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.508677 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.508681 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.508685 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.508689 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.508695 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.508700 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.508704 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.508708 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.508712 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.508716 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.508721 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.508725 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.508729 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.508733 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.508738 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.508742 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.508746 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.508750 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.508755 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.508759 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.508763 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.508767 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.508772 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.508776 | controller |       09:39:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.508780 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.508784 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.508789 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.508793 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.508799 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.508804 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.508808 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.508812 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.508816 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.508829 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.508833 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.508838 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.508842 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.508846 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.508850 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.508855 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.508859 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.508863 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.508867 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.508871 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.508876 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.508880 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.508884 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.508888 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.508893 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.508897 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.508901 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.508905 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.508910 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.508914 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.508918 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.508922 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.508926 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.508931 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.508935 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.508939 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.508943 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.508949 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.508954 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.508958 | controller |       09:39:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.508962 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.508966 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.508971 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.508975 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.508979 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.508983 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.508988 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.508992 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.508996 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.509000 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.509005 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.509014 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.509018 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.509023 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.509027 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.509031 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.509036 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.509040 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.509044 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.509048 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.509053 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.509057 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.509061 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.509065 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.509070 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.509076 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.509080 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.509084 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.509089 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.509095 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.509100 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.509105 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.509109 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.509113 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.509117 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.509122 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.509126 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.509130 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.509134 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.509139 | controller |       09:39:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.509143 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.509147 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.509151 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.509156 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.509160 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.509164 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.509168 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.509172 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.509177 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.509181 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.509185 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.509189 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.509199 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.509203 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.509208 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.509212 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.509216 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.509221 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.509225 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.509230 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.509238 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.509244 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.509250 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.509256 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.509261 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.509267 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.509273 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.509278 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.509282 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.509286 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.509290 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.509294 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.509299 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.509303 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.509307 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.509312 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.509316 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.509320 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.509324 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.509329 | controller |       09:39:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.509333 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.509337 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.509341 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.509345 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.509350 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.509354 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.509358 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.509362 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.509369 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.509373 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.509377 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.509382 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.509386 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.509392 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.509403 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.509407 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.509412 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.509416 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.509420 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.509425 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.509429 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.509433 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.509437 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.509442 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.509446 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.509450 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.509454 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.509458 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.509463 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.509467 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.509471 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.509475 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.509480 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.509484 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.509488 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.509492 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.509496 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.509501 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.509505 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.509509 | controller |       09:39:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.509513 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.509518 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.509522 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.509526 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.509534 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.509538 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.509543 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.509547 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.509551 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.509555 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.509560 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.509564 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.509568 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.509572 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.509576 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.509586 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.509590 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.509614 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.509622 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.509626 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.509633 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.509638 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.509642 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.509646 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.509652 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.509656 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.509660 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.509665 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.509669 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.509673 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.509677 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.509682 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.509686 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.509690 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.509694 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.509699 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.509703 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.509709 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.509713 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.509718 | controller |       09:39:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.509722 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.509726 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.509730 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.509735 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.509739 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.509743 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.509747 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.509752 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.509756 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.509760 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.509764 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.509769 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.509773 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.509777 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.509781 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.509785 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.509790 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.509800 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.509805 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.509809 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.509813 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.509818 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.509822 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.509826 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.509830 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.509835 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.509839 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.509843 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.509847 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.509854 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.509858 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.509862 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.509866 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.509871 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.509875 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.509879 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.509883 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.509888 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.509892 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.509896 | controller |       09:39:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.509900 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.509905 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.509909 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.509915 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.509919 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.509924 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.509928 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.509932 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.509936 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.509941 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.509945 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.509949 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.509953 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.509957 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.509962 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.509969 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.509974 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.509978 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.509987 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.509992 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.509998 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.510002 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.510006 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.510011 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.510015 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.510019 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.510023 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.510028 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.510032 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.510036 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.510040 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.510045 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.510049 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.510053 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.510058 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.510062 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.510066 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.510070 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.510075 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.510079 | controller |       09:39:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.510083 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.510087 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.510092 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.510096 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.510100 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.510104 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.510109 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.510113 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.510117 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.510121 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.510126 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.510130 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.510134 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.510140 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.510144 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.510149 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.510153 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.510157 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.510162 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.510171 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.510175 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.510180 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.510184 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.510190 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.510195 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.510199 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.510203 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.510207 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.510211 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.510216 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.510220 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.510224 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.510228 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.510234 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.510239 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.510243 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.510247 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.510251 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.510257 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.510261 | controller |       09:39:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.510266 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.510270 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.510274 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.510278 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.510284 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.510289 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.510293 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.510297 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.510301 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.510306 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.510310 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.510314 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.510319 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.510325 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.510331 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.510337 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.510343 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.510349 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.510354 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.510360 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.510370 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.510375 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.510379 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.510383 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.510388 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.510392 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.510396 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.510401 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.510405 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.510409 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.510416 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.510421 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.510426 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.510430 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.510434 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.510438 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.510443 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.510450 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.510454 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.510458 | controller |       09:39:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.510463 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.510467 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.510473 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.510477 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.510481 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.510485 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.510490 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.510494 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.510498 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.510502 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.510507 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.510511 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.510515 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.510519 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.510523 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.510528 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.510532 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.510536 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.510540 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.510544 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.510550 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.510555 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.510565 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.510569 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.510573 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.510578 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.510582 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.510586 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.510590 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.510611 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.510617 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.510621 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.510625 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.510630 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.510634 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.510638 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.510642 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.510647 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.510651 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.510655 | controller |       09:39:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.510659 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.510664 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.510668 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.510672 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.510676 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.510681 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.510685 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.510689 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.510693 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.510698 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.510702 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.510706 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.510710 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.510715 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.510719 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.510723 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.510727 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.510731 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.510736 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.510740 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.510746 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.510752 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.510756 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.510766 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.510770 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.510775 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.510779 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.510783 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.510788 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.510792 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.510796 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.510800 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.510805 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.510809 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.510813 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.510817 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.510822 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.510826 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.510830 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.510834 | controller |       09:39:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.510839 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.510843 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.510847 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.510851 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.510859 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.510863 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.510867 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.510872 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.510876 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.510891 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.510897 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.510901 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.510906 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.510912 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.510916 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.510921 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.510925 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.510929 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.510933 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.510938 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.510942 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.510946 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.510950 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.510955 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.510959 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.510969 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.510973 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.510977 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.510982 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.510986 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.510990 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.510994 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.510999 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.511003 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.511007 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.511011 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.511016 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.511020 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.511024 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.511028 | controller |       09:39:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.511033 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.511039 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.511043 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.511047 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.511053 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.511058 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.511062 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.511066 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.511070 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.511075 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.511079 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.511083 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.511087 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.511092 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.511096 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.511100 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.511104 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.511109 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.511113 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.511117 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.511121 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.511126 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.511130 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.511134 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.511138 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.511143 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.511152 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.511157 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.511161 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.511165 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.511169 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.511174 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.511178 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.511182 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.511186 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.511191 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.511195 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.511201 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.511206 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.511210 | controller |       09:39:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.511214 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.511218 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.511222 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.511227 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.511231 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.511235 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.511239 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.511244 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.511248 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.511252 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.511256 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.511260 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.511265 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.511269 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.511273 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.511279 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.511285 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.511289 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.511293 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.511298 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.511302 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.511306 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.511310 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.511315 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.511319 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.511323 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.511327 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.511331 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.511341 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.511347 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.511351 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.511355 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.511360 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.511364 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.511368 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.511372 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.511377 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.511381 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.511385 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.511389 | controller |       09:39:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.511394 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.511398 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.511402 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.511406 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.511411 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.511415 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.511419 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.511423 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.511429 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.511434 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.511441 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.511446 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.511452 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.511456 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.511460 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.511465 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.511469 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.511473 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.511477 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.511481 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.511490 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.511494 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.511498 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.511503 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.511507 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.511511 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.511515 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.511520 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.511524 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.511534 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.511538 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.511543 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.511547 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.511551 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.511556 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.511560 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.511564 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.511568 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.511573 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.511577 | controller |       09:39:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.511583 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.511588 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.511592 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.511609 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.511614 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.511618 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.511622 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.511626 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.511631 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.511635 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.511639 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.511643 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.511647 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.511654 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.511658 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.511662 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.511666 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.511671 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.511675 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.511679 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.511683 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.511688 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.511692 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.511696 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.511700 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.511704 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.511709 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.511713 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.511717 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.511721 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.511733 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.511738 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.511742 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.511746 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.511751 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.511755 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.511759 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.511763 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.511768 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.511772 | controller |       09:39:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.511776 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.511781 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.511785 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.511789 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.511795 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.511800 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.511804 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.511808 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.511812 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.511816 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.511821 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.511825 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.511829 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.511833 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.511839 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.511843 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.511848 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.511852 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.511856 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.511860 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.511865 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.511869 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.511873 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.511877 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.511882 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.511886 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.511890 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.511894 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.511899 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.511903 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.511907 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.511911 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.511921 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.511925 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.511929 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.511934 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.511938 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.511945 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.511950 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.511954 | controller |       09:39:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.511958 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.511962 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.511967 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.511971 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.511975 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.511979 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.511984 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.511988 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.511992 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.511996 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.512001 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.512005 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.512009 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.512013 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.512017 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.512022 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.512026 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.512030 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.512034 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.512039 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.512043 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.512047 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.512051 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.512056 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.512060 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.512064 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.512068 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.512073 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.512077 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.512083 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.512087 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.512091 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.512096 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.512105 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.512109 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.512113 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.512118 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.512122 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.512128 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.512133 | controller |       09:39:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.512137 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.512141 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.512146 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.512150 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.512156 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.512160 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.512164 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.512169 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.512173 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.512177 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.512181 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.512186 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.512190 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.512194 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.512198 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.512202 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.512207 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.512211 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.512215 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.512219 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.512226 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.512230 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.512234 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.512238 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.512243 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.512247 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.512251 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.512255 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.512259 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.512264 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.512268 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.512272 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.512277 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.512281 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.512290 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.512295 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.512299 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.512303 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.512308 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.512312 | controller |       09:39:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.512316 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.512320 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.512324 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.512329 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.512333 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.512339 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.512343 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.512347 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.512352 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.512356 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.512360 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.512364 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.512369 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.512375 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.512379 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.512383 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.512389 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.512393 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.512397 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.512402 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.512406 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.512412 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.512416 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.512420 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.512425 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.512429 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.512433 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.512437 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.512442 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.512446 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.512450 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.512454 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.512458 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.512463 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.512467 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.512471 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.512481 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.512485 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.512489 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.512493 | controller |       09:39:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.512500 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.512506 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.512512 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.512517 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.512524 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.512529 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.512533 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.512537 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.512541 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.512546 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.512550 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.512554 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.512558 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.512562 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.512567 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.512573 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.512577 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.512582 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.512586 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.512590 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.512605 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.512611 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.512616 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.512620 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.512624 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.512628 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.512632 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.512637 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.512641 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.512645 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.512649 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.512654 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.512658 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.512662 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.512666 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.512673 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.512677 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.512689 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.512694 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.512698 | controller |       09:39:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.512703 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.512707 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.512711 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.512715 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.512720 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.512724 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.512728 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.512733 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.512737 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.512741 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.512745 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.512749 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.512754 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.512758 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.512762 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.512766 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.512771 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.512775 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.512779 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.512783 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.512788 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.512792 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.512796 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.512800 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.512805 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.512809 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.512813 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.512817 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.512822 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.512828 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.512832 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.512836 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.512840 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.512845 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.512849 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.512853 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.512857 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.512862 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.512871 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.512876 | controller |       09:39:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.512880 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.512884 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.512888 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.512893 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.512897 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.512901 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.512905 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.512910 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.512914 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.512918 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.512922 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.512926 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.512933 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.512937 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.512941 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.512946 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.512950 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.512957 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.512962 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.512966 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.512972 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.512976 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.512981 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.512985 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.512989 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.512993 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.512997 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.513002 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.513006 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.513010 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.513016 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.513020 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.513024 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.513029 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.513033 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.513037 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.513041 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.513046 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.513050 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.513060 | controller |       09:39:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.513064 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.513068 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.513073 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.513077 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.513081 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.513086 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.513090 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.513094 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.513098 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.513103 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.513107 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.513111 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.513115 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.513122 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.513126 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.513130 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.513135 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.513139 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.513143 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.513147 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.513151 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.513156 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.513160 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.513164 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.513168 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.513173 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.513177 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.513181 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.513185 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.513189 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.513194 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.513198 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.513204 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.513208 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.513212 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.513217 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.513221 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.513225 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.513229 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.513233 | controller |       09:39:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.513238 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.513247 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.513252 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.513256 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.513263 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.513268 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.513272 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.513276 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.513280 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.513285 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.513289 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.513293 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.513297 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.513302 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.513306 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.513310 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.513314 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.513319 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.513323 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.513327 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.513333 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.513337 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.513342 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.513346 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.513350 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.513354 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.513359 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.513363 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.513367 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.513371 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.513376 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.513380 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.513384 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.513388 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.513393 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.513397 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.513401 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.513407 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.513411 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.513416 | controller |       09:39:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.513420 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.513424 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.513434 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.513438 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.513444 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.513449 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.513453 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.513457 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.513462 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.513466 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.513470 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.513479 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.513484 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.513488 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.513492 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.513496 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.513501 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.513505 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.513509 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.513513 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.513518 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.513522 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.513526 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.513530 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.513535 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.513539 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.513543 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.513547 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.513552 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.513558 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.513563 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.513569 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.513575 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.513581 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.513587 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.513592 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.513613 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.513617 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.513621 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.513626 | controller |       09:39:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.513630 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.513634 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.513639 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.513643 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.513654 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.513658 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.513662 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.513667 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.513671 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.513675 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.513680 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.513684 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.513688 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.513692 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.513697 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.513701 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.513705 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.513709 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.513714 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.513718 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.513725 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.513730 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.513734 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.513738 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.513742 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.513747 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.513751 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.513755 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.513761 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.513766 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.513770 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.513774 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.513778 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.513783 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.513787 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.513791 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.513795 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.513799 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.513804 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.513808 | controller |       09:39:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.513812 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.513816 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.513821 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.513825 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.513829 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.513839 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.513843 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.513847 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.513852 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.513856 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.513860 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.513865 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.513869 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.513875 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.513879 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.513885 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.513889 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.513894 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.513898 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.513902 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.513906 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.513911 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.513915 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.513919 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.513923 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.513928 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.513932 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.513936 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.513940 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.513945 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.513949 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.513953 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.513957 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.513962 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.513966 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.513970 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.513974 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.513979 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.513983 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.513987 | controller |       09:39:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.513991 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.513996 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.514000 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.514004 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.514011 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.514015 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.514025 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.514029 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.514034 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.514038 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.514044 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.514049 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.514053 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.514057 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.514061 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.514066 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.514070 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.514074 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.514079 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.514083 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.514087 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.514091 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.514096 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.514100 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.514104 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.514108 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.514113 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.514117 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.514121 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.514125 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.514130 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.514134 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.514138 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.514142 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.514146 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.514151 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.514155 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.514161 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.514166 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.514170 | controller |       09:39:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.514174 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:39:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.514178 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.514183 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.514187 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.514191 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.514195 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.514200 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.514204 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.514213 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.514218 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.514222 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.514226 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.514231 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.514235 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.514239 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.514243 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.514248 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.514252 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.514256 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.514260 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.514267 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.514271 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.514275 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.514280 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.514284 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.514288 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.514292 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.514298 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.514302 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.514309 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.514315 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.514319 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.514323 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.514327 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.514332 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.514336 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.514340 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.514344 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:39:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.514349 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.514353 | controller |       09:39:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.514357 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.514361 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.514366 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.514370 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.514374 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.514378 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.514382 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.514387 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.514391 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.514400 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.514405 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.514409 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.514413 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.514417 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.514422 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.514426 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.514430 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.514434 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.514439 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.514443 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.514449 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.514453 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.514458 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.514462 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.514466 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.514470 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.514475 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.514479 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.514483 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.514487 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.514492 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.514496 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.514500 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.514504 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.514509 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.514513 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.514517 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.514521 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.514527 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.514531 | controller |       09:40:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.514536 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.514540 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.514544 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.514548 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.514553 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.514557 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.514561 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.514565 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.514570 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.514576 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.514580 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.514589 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.514605 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.514613 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.514618 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.514622 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.514626 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.514630 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.514635 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.514639 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.514643 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.514647 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.514653 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.514659 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.514665 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.514671 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.514676 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.514682 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.514688 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.514694 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.514700 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.514704 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.514708 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.514713 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.514717 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.514721 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.514725 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.514730 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.514734 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.514738 | controller |       09:40:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.514743 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.514747 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.514751 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.514755 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.514764 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.514768 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.514773 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.514777 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.514781 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.514785 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.514790 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.514794 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.514805 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.514810 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.514814 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.514819 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.514823 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.514827 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.514831 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.514836 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.514840 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.514844 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.514848 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.514853 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.514857 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.514863 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.514867 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.514872 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.514876 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.514880 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.514897 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.514903 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.514907 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.514911 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.514916 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.514920 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.514924 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.514931 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.514935 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.514939 | controller |       09:40:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.514943 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.514948 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.514952 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.514956 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.514960 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.514965 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.514969 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.514973 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.514977 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.514982 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.514986 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.514990 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.514994 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.514998 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.515008 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.515012 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.515017 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.515021 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.515025 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.515029 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.515034 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.515038 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.515042 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.515046 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.515051 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.515055 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.515059 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.515063 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.515067 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.515075 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.515079 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.515083 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.515088 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.515092 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.515096 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.515100 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.515105 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.515109 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.515113 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.515117 | controller |       09:40:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.515121 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.515126 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.515130 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.515134 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.515138 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.515143 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.515147 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.515151 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.515157 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.515161 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.515166 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.515170 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.515174 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.515178 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.515183 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.515194 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.515199 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.515203 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.515208 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.515212 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.515219 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.515224 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.515228 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.515232 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.515236 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.515241 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.515245 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.515249 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.515253 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.515258 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.515262 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.515266 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.515270 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.515275 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.515279 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.515283 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.515287 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.515292 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.515296 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.515300 | controller |       09:40:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.515304 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.515309 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.515313 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.515317 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.515321 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.515326 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.515330 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.515334 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.515338 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.515343 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.515347 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.515351 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.515355 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.515361 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.515366 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.515370 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.515374 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.515383 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.515388 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.515392 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.515396 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.515401 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.515405 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.515409 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.515415 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.515419 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.515424 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.515428 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.515432 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.515436 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.515440 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.515445 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.515449 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.515453 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.515457 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.515462 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.515466 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.515470 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.515474 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.515479 | controller |       09:40:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.515483 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.515487 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.515491 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.515496 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.515501 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.515506 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.515510 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.515514 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.515519 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.515523 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.515527 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.515531 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.515535 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.515540 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.515544 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.515548 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.515552 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.515557 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.515566 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.515570 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.515574 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.515579 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.515583 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.515587 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.515591 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.515611 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.515616 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.515620 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.515624 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.515629 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.515635 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.515639 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.515643 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.515648 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.515652 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.515656 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.515660 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.515666 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.515671 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.515675 | controller |       09:40:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.515679 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.515684 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.515688 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.515694 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.515698 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.515702 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.515707 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.515711 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.515715 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.515719 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.515724 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.515728 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.515732 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.515736 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.515741 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.515745 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.515749 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.515753 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.515757 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.515768 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.515772 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.515777 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.515781 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.515785 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.515789 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.515794 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.515799 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.515804 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.515810 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.515818 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.515824 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.515829 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.515833 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.515837 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.515841 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.515846 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.515850 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.515854 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.515858 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.515863 | controller |       09:40:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.515867 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.515871 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.515875 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.515879 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.515884 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.515888 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.515892 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.515896 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.515901 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.515905 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.515909 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.515913 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.515917 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.515921 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.515926 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.515930 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.515934 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.515938 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.515943 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.515947 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.515959 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.515963 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.515968 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.515974 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.515978 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.515983 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.515987 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.515991 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.515995 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.516000 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.516004 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.516008 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.516012 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.516017 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.516021 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.516025 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.516029 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.516035 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.516040 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.516044 | controller |       09:40:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.516048 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.516053 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.516057 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.516061 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.516069 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.516073 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.516077 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.516082 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.516086 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.516090 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.516094 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.516098 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.516103 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.516109 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.516113 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.516117 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.516121 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.516126 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.516130 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.516134 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.516140 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.516145 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.516154 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.516159 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.516163 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.516167 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.516172 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.516176 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.516180 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.516184 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.516189 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.516193 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.516197 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.516202 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.516206 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.516210 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.516214 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.516219 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.516223 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.516227 | controller |       09:40:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.516231 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.516236 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.516242 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.516246 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.516252 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.516256 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.516260 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.516265 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.516269 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.516273 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.516277 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.516282 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.516286 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.516290 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.516294 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.516299 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.516303 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.516307 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.516311 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.516316 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.516320 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.516324 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.516328 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.516338 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.516342 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.516346 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.516351 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.516355 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.516359 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.516363 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.516368 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.516372 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.516376 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.516381 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.516385 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.516389 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.516393 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.516399 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.516404 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.516408 | controller |       09:40:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.516412 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.516416 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.516421 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.516425 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.516429 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.516433 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.516437 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.516442 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.516446 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.516450 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.516454 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.516459 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.516463 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.516467 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.516471 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.516477 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.516481 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.516486 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.516490 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.516494 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.516498 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.516504 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.516508 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.516513 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.516522 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.516526 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.516530 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.516534 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.516539 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.516545 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.516549 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.516553 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.516558 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.516562 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.516566 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.516570 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.516575 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.516579 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.516583 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.516587 | controller |       09:40:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.516591 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.516609 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.516613 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.516617 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.516621 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.516628 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.516632 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.516636 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.516640 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.516645 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.516649 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.516653 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.516657 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.516661 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.516665 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.516670 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.516674 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.516678 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.516682 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.516686 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.516692 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.516697 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.516701 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.516705 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.516709 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.516714 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.516724 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.516729 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.516733 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.516737 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.516741 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.516746 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.516750 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.516754 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.516758 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.516763 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.516767 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.516771 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.516775 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.516779 | controller |       09:40:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.516784 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.516790 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.516794 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.516798 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.516802 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.516807 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.516811 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.516815 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.516819 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.516823 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.516828 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.516832 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.516836 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.516842 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.516847 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.516851 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.516855 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.516859 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.516864 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.516868 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.516872 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.516876 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.516880 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.516885 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.516889 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.516893 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.516897 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.516907 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.516911 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.516916 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.516922 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.516926 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.516930 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.516935 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.516939 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.516943 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.516947 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.516952 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.516956 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.516960 | controller |       09:40:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.516964 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.516969 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.516973 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.516977 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.516983 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.516987 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.516992 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.516996 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.517000 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.517004 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.517009 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.517013 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.517017 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.517021 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.517026 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.517030 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.517037 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.517041 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.517045 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.517050 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.517055 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.517060 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.517064 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.517068 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.517072 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.517077 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.517081 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.517085 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.517089 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.517099 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.517103 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.517107 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.517112 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.517116 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.517120 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.517124 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.517129 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.517135 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.517139 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.517143 | controller |       09:40:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.517148 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.517152 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.517156 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.517160 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.517164 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.517169 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.517173 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.517177 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.517181 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.517185 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.517190 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.517194 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.517198 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.517202 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.517206 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.517211 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.517215 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.517219 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.517223 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.517228 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.517232 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.517236 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.517240 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.517245 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.517249 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.517253 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.517257 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.517262 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.517266 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.517273 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.517283 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.517287 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.517292 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.517296 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.517300 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.517305 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.517309 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.517313 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.517317 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.517322 | controller |       09:40:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.517328 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.517332 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.517336 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.517340 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.517346 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.517350 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.517355 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.517359 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.517363 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.517367 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.517371 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.517375 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.517380 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.517384 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.517388 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.517392 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.517396 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.517401 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.517405 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.517409 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.517415 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.517419 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.517424 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.517428 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.517432 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.517436 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.517441 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.517445 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.517449 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.517453 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.517458 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.517467 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.517472 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.517476 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.517480 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.517485 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.517489 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.517493 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.517497 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.517502 | controller |       09:40:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.517506 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.517510 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.517514 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.517519 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.517523 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.517527 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.517531 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.517536 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.517540 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.517544 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.517548 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.517553 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.517557 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.517563 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.517569 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.517573 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.517577 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.517582 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.517586 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.517590 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.517605 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.517611 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.517615 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.517620 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.517624 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.517628 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.517632 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.517636 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.517641 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.517645 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.517649 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.517654 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.517658 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.517668 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.517672 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.517676 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.517681 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.517685 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.517690 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.517694 | controller |       09:40:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.517698 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.517702 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.517707 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.517711 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.517717 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.517721 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.517726 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.517730 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.517734 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.517738 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.517743 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.517747 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.517751 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.517755 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.517760 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.517765 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.517770 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.517774 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.517778 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.517783 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.517787 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.517791 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.517795 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.517799 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.517804 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.517808 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.517812 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.517816 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.517821 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.517825 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.517829 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.517833 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.517838 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.517842 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.517851 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.517855 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.517860 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.517866 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.517872 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.517876 | controller |       09:40:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.517880 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.517885 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.517889 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.517893 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.517897 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.517902 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.517906 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.517910 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.517914 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.517918 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.517923 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.517927 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.517931 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.517935 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.517940 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.517944 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.517948 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.517952 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.517957 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.517961 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.517966 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.517971 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.517975 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.517979 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.517983 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.517988 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.517992 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.517996 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.518000 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.518006 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.518010 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.518015 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.518019 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.518023 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.518027 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.518037 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.518041 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.518045 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.518049 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.518054 | controller |       09:40:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.518058 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.518062 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.518066 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.518071 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.518075 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.518079 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.518084 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.518088 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.518092 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.518096 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.518101 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.518105 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.518109 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.518116 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.518121 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.518125 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.518129 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.518133 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.518138 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.518142 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.518148 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.518152 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.518156 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.518160 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.518165 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.518169 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.518173 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.518177 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.518182 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.518186 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.518192 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.518196 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.518200 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.518205 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.518209 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.518213 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.518217 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.518227 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.518231 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.518235 | controller |       09:40:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.518240 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.518244 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.518248 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.518253 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.518257 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.518261 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.518265 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.518270 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.518274 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.518278 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.518282 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.518287 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.518291 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.518297 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.518301 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.518306 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.518310 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.518314 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.518318 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.518323 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.518327 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.518331 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.518335 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.518340 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.518344 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.518348 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.518352 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.518357 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.518361 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.518365 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.518369 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.518374 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.518378 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.518382 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.518386 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.518392 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.518396 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.518401 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.518410 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.518414 | controller |       09:40:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.518419 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.518423 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.518427 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.518431 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.518438 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.518442 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.518446 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.518450 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.518455 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.518459 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.518463 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.518468 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.518472 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.518476 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.518480 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.518485 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.518489 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.518493 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.518497 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.518502 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.518506 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.518510 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.518514 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.518518 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.518523 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.518527 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.518531 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.518535 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.518540 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.518544 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.518548 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.518552 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.518556 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.518561 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.518565 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.518569 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.518573 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.518579 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.518583 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.518604 | controller |       09:40:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.518611 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.518615 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.518619 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.518623 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.518628 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.518632 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.518636 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.518640 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.518645 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.518649 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.518653 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.518657 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.518663 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.518667 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.518672 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.518676 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.518680 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.518684 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.518689 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.518693 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.518697 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.518701 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.518706 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.518710 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.518714 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.518718 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.518722 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.518727 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.518731 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.518737 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.518741 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.518745 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.518750 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.518754 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.518758 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.518763 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.518767 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.518771 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.518777 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.518781 | controller |       09:40:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.518786 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.518795 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.518800 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.518804 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.518808 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.518812 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.518817 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.518821 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.518825 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.518829 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.518833 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.518838 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.518842 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.518846 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.518850 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.518855 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.518859 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.518863 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.518867 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.518871 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.518879 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.518884 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.518903 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.518908 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.518912 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.518916 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.518920 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.518925 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.518929 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.518933 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.518937 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.518942 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.518948 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.518952 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.518956 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.518961 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.518965 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.518969 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.518973 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.518978 | controller |       09:40:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.518982 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.518986 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.518996 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.519001 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.519007 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.519013 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.519019 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.519025 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.519031 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.519037 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.519043 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.519047 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.519051 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.519058 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.519062 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.519070 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.519074 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.519079 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.519083 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.519087 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.519091 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.519096 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.519100 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.519104 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.519108 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.519113 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.519117 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.519121 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.519125 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.519130 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.519134 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.519138 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.519143 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.519147 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.519151 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.519155 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.519160 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.519164 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.519168 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.519172 | controller |       09:40:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.519177 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.519181 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.519185 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.519190 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.519202 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.519206 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.519211 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.519215 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.519219 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.519223 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.519228 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.519234 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.519238 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.519242 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.519247 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.519251 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.519255 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.519259 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.519264 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.519268 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.519272 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.519276 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.519281 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.519285 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.519289 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.519293 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.519298 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.519302 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.519306 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.519310 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.519315 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.519319 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.519323 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.519327 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.519332 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.519336 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.519340 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.519346 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.519351 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.519355 | controller |       09:40:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.519359 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.519363 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.519368 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.519372 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.519376 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.519385 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.519390 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.519394 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.519398 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.519403 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.519407 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.519411 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.519415 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.519420 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.519424 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.519428 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.519433 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.519439 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.519443 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.519447 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.519452 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.519456 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.519460 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.519464 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.519468 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.519473 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.519477 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.519481 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.519487 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.519493 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.519498 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.519503 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.519507 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.519511 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.519515 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.519520 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.519524 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.519528 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.519532 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.519537 | controller |       09:40:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.519541 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.519545 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.519549 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.519554 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.519558 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.519562 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.519571 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.519576 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.519580 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.519584 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.519588 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.519593 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.519610 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.519614 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.519619 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.519623 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.519627 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.519631 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.519635 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.519640 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.519646 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.519650 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.519654 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.519659 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.519663 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.519667 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.519671 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.519675 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.519680 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.519684 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.519688 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.519692 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.519697 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.519701 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.519705 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.519709 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.519714 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.519718 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.519722 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.519726 | controller |       09:40:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.519731 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.519735 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.519739 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.519743 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.519748 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.519752 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.519756 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.519760 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.519770 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.519775 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.519781 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.519785 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.519790 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.519796 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.519800 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.519804 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.519808 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.519813 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.519817 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.519821 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.519827 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.519831 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.519836 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.519840 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.519844 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.519848 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.519852 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.519857 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.519861 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.519865 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.519869 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.519874 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.519878 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.519882 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.519886 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.519891 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.519895 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.519899 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.519903 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.519909 | controller |       09:40:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.519914 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.519918 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.519922 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.519926 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.519934 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.519938 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.519942 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.519947 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.519951 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.519961 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.519965 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.519969 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.519974 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.519978 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.519982 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.519986 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.519991 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.519995 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.519999 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.520004 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.520008 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.520012 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.520016 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.520020 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.520025 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.520029 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.520033 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.520040 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.520045 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.520049 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.520053 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.520057 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.520062 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.520066 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.520070 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.520074 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.520079 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.520085 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.520089 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.520094 | controller |       09:40:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.520100 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.520106 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.520112 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.520117 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.520122 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.520126 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.520130 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.520134 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.520139 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.520143 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.520147 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.520158 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.520162 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.520166 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.520171 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.520175 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.520179 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.520184 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.520188 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.520192 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.520196 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.520201 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.520205 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.520209 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.520213 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.520218 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.520222 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.520226 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.520230 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.520237 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.520241 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.520245 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.520249 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.520254 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.520258 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.520262 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.520266 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.520271 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.520275 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.520279 | controller |       09:40:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.520283 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.520288 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.520292 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.520296 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.520300 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.520305 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.520309 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.520313 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.520317 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.520324 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.520328 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.520332 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.520342 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.520346 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.520350 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.520356 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.520361 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.520365 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.520369 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.520374 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.520380 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.520384 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.520388 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.520392 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.520397 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.520401 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.520405 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.520409 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.520413 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.520418 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.520422 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.520426 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.520430 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.520435 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.520439 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.520443 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.520447 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.520452 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.520456 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.520460 | controller |       09:40:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.520464 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.520468 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.520473 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.520477 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.520481 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.520485 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.520490 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.520494 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.520498 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.520502 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.520506 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.520511 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.520515 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.520521 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.520530 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.520535 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.520539 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.520543 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.520547 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.520552 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.520556 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.520561 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.520565 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.520569 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.520573 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.520580 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.520584 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.520588 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.520592 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.520609 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.520614 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.520618 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.520622 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.520627 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.520631 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.520635 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.520640 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.520644 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.520648 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.520652 | controller |       09:40:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.520657 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.520661 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.520665 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.520669 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.520675 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.520680 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.520684 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.520688 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.520693 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.520697 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.520701 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.520705 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.520710 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.520714 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.520718 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.520728 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.520733 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.520737 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.520741 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.520746 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.520752 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.520756 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.520761 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.520765 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.520769 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.520773 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.520777 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.520782 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.520786 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.520790 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.520796 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.520800 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.520805 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.520809 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.520813 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.520817 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.520821 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.520828 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.520832 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.520836 | controller |       09:40:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.520840 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.520845 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.520849 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.520853 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.520857 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.520863 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.520867 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.520872 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.520876 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.520880 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.520885 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.520889 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.520893 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.520897 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.520901 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.520906 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.520917 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.520921 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.520926 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.520930 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.520934 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.520939 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.520943 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.520947 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.520951 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.520956 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.520960 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.520964 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.520968 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.520975 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.520979 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.520983 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.520988 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.520992 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.520996 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.521000 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.521005 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.521009 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.521013 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.521017 | controller |       09:40:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.521022 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.521026 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.521030 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.521034 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.521038 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.521043 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.521047 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.521051 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.521055 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.521060 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.521064 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.521068 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.521072 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.521077 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.521081 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.521085 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.521089 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.521094 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.521103 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.521108 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.521115 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.521119 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.521124 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.521128 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.521134 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.521138 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.521143 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.521147 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.521151 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.521155 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.521159 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.521164 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.521168 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.521174 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.521180 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.521186 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.521192 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.521198 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.521204 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.521210 | controller |       09:40:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.521215 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.521219 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.521223 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.521228 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.521234 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.521239 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.521243 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.521247 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.521251 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.521256 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.521260 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.521264 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.521268 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.521274 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.521279 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.521283 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.521287 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.521291 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.521296 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.521307 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.521311 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.521315 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.521320 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.521324 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.521329 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.521333 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.521337 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.521341 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.521346 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.521350 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.521354 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.521358 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.521363 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.521367 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.521371 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.521375 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.521380 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.521384 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.521388 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.521393 | controller |       09:40:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.521397 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.521401 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.521405 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.521412 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.521418 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.521422 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.521426 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.521430 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.521435 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.521439 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.521443 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.521447 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.521451 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.521456 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.521460 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.521464 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.521468 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.521473 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.521477 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.521481 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.521491 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.521495 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.521499 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.521503 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.521508 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.521512 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.521517 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.521521 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.521525 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.521529 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.521533 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.521538 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.521542 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.521546 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.521550 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.521555 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.521559 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.521565 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.521569 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.521574 | controller |       09:40:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.521578 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.521582 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.521586 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.521591 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.521605 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.521612 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.521616 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.521620 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.521624 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.521629 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.521633 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.521637 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.521641 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.521646 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.521650 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.521656 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.521660 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.521664 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.521669 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.521673 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.521679 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.521688 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.521693 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.521699 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.521703 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.521707 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.521712 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.521716 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.521720 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.521726 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.521730 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.521735 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.521739 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.521743 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.521747 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.521752 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.521756 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.521760 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.521764 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.521769 | controller |       09:40:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.521773 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.521777 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.521781 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.521786 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.521790 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.521794 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.521798 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.521803 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.521807 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.521811 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.521815 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.521820 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.521824 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.521828 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.521832 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.521837 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.521841 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.521845 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.521849 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.521854 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.521860 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.521864 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.521868 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.521878 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.521882 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.521886 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.521891 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.521895 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.521899 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.521904 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.521908 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.521912 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.521917 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.521921 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.521925 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.521929 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.521934 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.521938 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.521942 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.521946 | controller |       09:40:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.521951 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.521955 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.521961 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.521965 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.521970 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.521974 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.521978 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.521982 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.521987 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.521991 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.521995 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.521999 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.522003 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.522010 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.522014 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.522018 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.522023 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.522027 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.522031 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.522036 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.522040 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.522044 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.522048 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.522053 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.522062 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.522067 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.522071 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.522075 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.522080 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.522084 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.522094 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.522099 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.522103 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.522108 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.522112 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.522116 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.522121 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.522125 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.522129 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.522134 | controller |       09:40:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.522138 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.522142 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.522147 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.522151 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.522158 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.522163 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.522167 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.522172 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.522176 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.522180 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.522185 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.522189 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.522193 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.522197 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.522202 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.522206 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.522210 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.522215 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.522219 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.522223 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.522228 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.522234 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.522238 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.522242 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.522247 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.522251 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.522261 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.522265 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.522270 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.522274 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.522280 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.522286 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.522292 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.522298 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.522304 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.522309 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.522315 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.522324 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.522330 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.522335 | controller |       09:40:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.522340 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.522344 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.522349 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.522353 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.522357 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.522362 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.522366 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.522370 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.522375 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.522379 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.522383 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.522388 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.522392 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.522396 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.522400 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.522405 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.522409 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.522413 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.522418 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.522422 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.522426 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.522431 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.522435 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.522439 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.522443 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.522448 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.522452 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.522462 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.522467 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.522474 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.522478 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.522483 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.522487 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.522491 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.522496 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.522500 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.522504 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.522509 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.522513 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.522517 | controller |       09:40:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.522522 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.522528 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.522533 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.522537 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.522543 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.522547 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.522552 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.522556 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.522560 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.522565 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.522569 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.522573 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.522578 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.522582 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.522586 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.522590 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.522608 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.522616 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.522622 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.522628 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.522639 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.522645 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.522651 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.522657 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.522663 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.522669 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.522673 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.522677 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.522682 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.522693 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.522698 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.522702 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.522707 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.522711 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.522716 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.522720 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.522724 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.522729 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.522733 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.522737 | controller |       09:40:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.522742 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.522746 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.522750 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.522755 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.522759 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.522763 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.522768 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.522772 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.522776 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.522780 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.522785 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.522789 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.522793 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.522800 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.522804 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.522808 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.522815 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.522819 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.522824 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.522828 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.522833 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.522837 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.522841 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.522846 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.522850 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.522854 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.522858 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.522863 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.522867 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.522871 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.522881 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.522900 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.522906 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.522911 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.522915 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.522920 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.522924 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.522929 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.522933 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.522937 | controller |       09:40:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.522942 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.522946 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.522950 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.522955 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.522961 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.522966 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.522970 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.522974 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.522979 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.522983 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.522987 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.522992 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.522996 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.523000 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.523004 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.523011 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.523015 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.523020 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.523024 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.523028 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.523033 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.523037 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.523041 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.523046 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.523050 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.523054 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.523058 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.523063 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.523067 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.523071 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.523076 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.523086 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.523090 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.523095 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.523099 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.523103 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.523108 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.523114 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.523120 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.523124 | controller |       09:40:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.523130 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.523134 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.523139 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.523143 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.523147 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.523152 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.523156 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.523160 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.523165 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.523169 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.523173 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.523177 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.523182 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.523186 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.523190 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.523194 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.523199 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.523203 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.523207 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.523212 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.523216 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.523220 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.523225 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.523229 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.523233 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.523238 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.523242 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.523246 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.523250 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.523258 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.523262 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.523267 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.523271 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.523281 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.523286 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.523290 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.523294 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.523299 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.523303 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.523307 | controller |       09:40:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.523312 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.523316 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.523320 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.523325 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.523329 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.523334 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.523338 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.523342 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.523346 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.523353 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.523357 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.523362 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.523366 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.523370 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.523377 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.523383 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.523389 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.523395 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.523401 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.523405 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.523412 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.523416 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.523420 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.523425 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.523429 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.523433 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.523438 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.523442 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.523446 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.523451 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.523457 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.523461 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.523466 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.523470 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.523480 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.523485 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.523489 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.523493 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.523498 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.523502 | controller |       09:40:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.523507 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.523511 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.523515 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.523520 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.523524 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.523528 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.523533 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.523537 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.523541 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.523546 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.523550 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.523555 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.523559 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.523565 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.523569 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.523574 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.523578 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.523582 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.523587 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.523591 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.523608 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.523617 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.523621 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.523625 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.523630 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.523634 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.523638 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.523643 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.523647 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.523651 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.523656 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.523660 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.523664 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.523669 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.523673 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.523683 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.523688 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.523692 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.523698 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.523703 | controller |       09:40:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.523707 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:40:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.523712 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.523716 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.523720 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.523726 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.523731 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.523735 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.523739 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.523744 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.523748 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.523752 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.523757 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.523761 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.523765 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.523770 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.523774 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.523778 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.523783 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.523787 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.523791 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.523796 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.523800 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.523804 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.523809 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.523813 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.523817 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.523822 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.523826 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.523830 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.523834 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.523839 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.523843 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.523847 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.523852 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.523856 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.523860 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.523865 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.523876 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:40:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.523881 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.523885 | controller |       09:40:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.523889 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.523894 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.523898 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.523903 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.523909 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.523913 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.523917 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.523922 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.523926 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.523930 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.523935 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.523939 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.523943 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.523949 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.523954 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.523958 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.523962 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.523967 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.523971 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.523975 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.523979 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.523984 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.523988 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.523992 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.523997 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.524001 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.524005 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.524010 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.524014 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.524020 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.524024 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.524029 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.524033 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.524037 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.524042 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.524046 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.524050 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.524055 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.524064 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.524069 | controller |       09:41:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.524073 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.524078 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.524082 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.524086 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.524091 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.524095 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.524099 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.524104 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.524108 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.524112 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.524117 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.524121 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.524125 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.524130 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.524134 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.524138 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.524142 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.524147 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.524151 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.524155 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.524161 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.524166 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.524170 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.524174 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.524178 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.524182 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.524187 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.524191 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.524195 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.524199 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.524204 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.524208 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.524212 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.524216 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.524221 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.524227 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.524231 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.524235 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.524240 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.524249 | controller |       09:41:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.524253 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.524258 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.524262 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.524266 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.524271 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.524275 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.524279 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.524283 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.524288 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.524292 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.524296 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.524300 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.524305 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.524312 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.524316 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.524322 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.524327 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.524331 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.524335 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.524339 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.524344 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.524348 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.524352 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.524356 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.524361 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.524365 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.524369 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.524373 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.524377 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.524382 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.524386 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.524390 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.524394 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.524399 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.524403 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.524407 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.524411 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.524416 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.524420 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.524424 | controller |       09:41:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.524434 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.524438 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.524443 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.524449 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.524457 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.524463 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.524469 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.524475 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.524480 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.524486 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.524491 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.524496 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.524502 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.524507 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.524511 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.524515 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.524519 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.524523 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.524528 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.524532 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.524538 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.524542 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.524546 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.524550 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.524555 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.524559 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.524563 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.524567 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.524572 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.524576 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.524580 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.524584 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.524588 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.524593 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.524610 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.524615 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.524619 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.524626 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.524630 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.524634 | controller |       09:41:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.524638 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.524643 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.524653 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.524657 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.524662 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.524666 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.524670 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.524674 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.524679 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.524683 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.524687 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.524692 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.524696 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.524700 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.524704 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.524709 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.524713 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.524717 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.524721 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.524726 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.524730 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.524734 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.524738 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.524744 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.524749 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.524753 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.524757 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.524761 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.524766 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.524771 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.524777 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.524782 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.524787 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.524792 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.524796 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.524800 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.524804 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.524809 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.524813 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.524817 | controller |       09:41:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.524821 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.524825 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.524830 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.524839 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.524844 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.524848 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.524852 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.524856 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.524861 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.524865 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.524869 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.524874 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.524878 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.524882 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.524886 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.524891 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.524895 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.524899 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.524903 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.524907 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.524913 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.524918 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.524922 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.524926 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.524930 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.524935 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.524939 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.524943 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.524947 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.524951 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.524956 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.524960 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.524964 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.524969 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.524973 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.524977 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.524981 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.524985 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.524990 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.524994 | controller |       09:41:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.524998 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.525003 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.525007 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.525011 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.525015 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.525025 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.525029 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.525033 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.525038 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.525042 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.525046 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.525052 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.525057 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.525063 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.525067 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.525071 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.525075 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.525080 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.525084 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.525088 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.525092 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.525097 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.525101 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.525105 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.525109 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.525114 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.525118 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.525122 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.525126 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.525130 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.525135 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.525139 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.525143 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.525147 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.525152 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.525156 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.525160 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.525164 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.525169 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.525173 | controller |       09:41:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.525177 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.525181 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.525186 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.525190 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.525198 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.525206 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.525216 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.525220 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.525225 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.525229 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.525233 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.525237 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.525242 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.525246 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.525250 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.525254 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.525259 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.525263 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.525267 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.525271 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.525276 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.525280 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.525284 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.525288 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.525293 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.525297 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.525301 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.525305 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.525309 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.525314 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.525320 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.525324 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.525328 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.525333 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.525337 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.525341 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.525345 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.525351 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.525356 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.525360 | controller |       09:41:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.525364 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.525368 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.525373 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.525377 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.525381 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.525385 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.525389 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.525399 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.525403 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.525407 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.525412 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.525416 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.525420 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.525424 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.525429 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.525433 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.525437 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.525441 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.525446 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.525450 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.525456 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.525460 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.525464 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.525469 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.525473 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.525477 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.525481 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.525486 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.525490 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.525496 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.525500 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.525504 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.525509 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.525513 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.525517 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.525521 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.525525 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.525530 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.525534 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.525538 | controller |       09:41:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.525542 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.525547 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.525551 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.525555 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.525559 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.525563 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.525568 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.525572 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.525576 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.525611 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.525621 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.525625 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.525629 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.525634 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.525638 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.525644 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.525648 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.525652 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.525657 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.525661 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.525667 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.525671 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.525676 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.525680 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.525684 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.525688 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.525692 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.525697 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.525701 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.525705 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.525709 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.525714 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.525718 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.525722 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.525726 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.525731 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.525735 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.525739 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.525743 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.525747 | controller |       09:41:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.525752 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.525756 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.525760 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.525764 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.525769 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.525773 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.525777 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.525781 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.525785 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.525790 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.525799 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.525804 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.525808 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.525814 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.525819 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.525823 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.525827 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.525831 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.525836 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.525840 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.525844 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.525848 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.525853 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.525857 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.525861 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.525865 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.525869 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.525875 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.525880 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.525884 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.525888 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.525892 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.525897 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.525901 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.525905 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.525909 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.525914 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.525918 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.525922 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.525926 | controller |       09:41:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.525931 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.525935 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.525939 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.525943 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.525949 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.525954 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.525958 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.525962 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.525966 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.525970 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.525975 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.525979 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.525988 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.525993 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.525997 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.526001 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.526005 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.526010 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.526014 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.526018 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.526022 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.526027 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.526031 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.526035 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.526039 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.526044 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.526048 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.526052 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.526056 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.526061 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.526067 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.526071 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.526075 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.526079 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.526084 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.526089 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.526095 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.526104 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.526110 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.526116 | controller |       09:41:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.526122 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.526126 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.526130 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.526135 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.526139 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.526143 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.526148 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.526154 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.526160 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.526168 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.526174 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.526178 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.526183 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.526193 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.526198 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.526202 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.526206 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.526211 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.526215 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.526220 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.526226 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.526232 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.526238 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.526243 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.526249 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.526254 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.526260 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.526266 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.526272 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.526280 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.526285 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.526289 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.526293 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.526298 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.526302 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.526306 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.526310 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.526315 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.526319 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.526323 | controller |       09:41:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.526327 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.526332 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.526336 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.526340 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.526344 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.526349 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.526353 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.526357 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.526361 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.526366 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.526370 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.526374 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.526378 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.526383 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.526387 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.526398 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.526402 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.526407 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.526411 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.526415 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.526426 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.526431 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.526435 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.526439 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.526445 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.526453 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.526459 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.526465 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.526470 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.526475 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.526479 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.526484 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.526488 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.526492 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.526496 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.526501 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.526505 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.526509 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.526513 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.526517 | controller |       09:41:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.526522 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.526526 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.526530 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.526534 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.526540 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.526545 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.526549 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.526553 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.526557 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.526562 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.526566 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.526570 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.526574 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.526581 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.526585 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.526589 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.526610 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.526618 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.526622 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.526627 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.526632 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.526638 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.526644 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.526650 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.526656 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.526660 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.526665 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.526669 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.526673 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.526677 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.526682 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.526686 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.526690 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.526695 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.526699 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.526703 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.526707 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.526712 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.526716 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.526720 | controller |       09:41:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.526724 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.526729 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.526733 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.526737 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.526744 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.526748 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.526752 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.526756 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.526763 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.526767 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.526772 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.526776 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.526780 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.526784 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.526788 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.526793 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.526797 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.526807 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.526812 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.526816 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.526821 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.526825 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.526829 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.526834 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.526838 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.526842 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.526846 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.526851 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.526855 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.526859 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.526864 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.526868 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.526872 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.526876 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.526881 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.526885 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.526902 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.526912 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.526917 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.526921 | controller |       09:41:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.526926 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.526930 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.526934 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.526938 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.526943 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.526947 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.526951 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.526955 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.526960 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.526964 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.526968 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.526972 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.526977 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.526981 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.526985 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.526991 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.526996 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.527000 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.527005 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.527017 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.527021 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.527025 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.527030 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.527034 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.527040 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.527044 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.527049 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.527053 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.527057 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.527064 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.527068 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.527072 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.527076 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.527081 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.527085 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.527089 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.527093 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.527098 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.527102 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.527106 | controller |       09:41:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.527111 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.527115 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.527119 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.527123 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.527128 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.527132 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.527136 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.527140 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.527145 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.527149 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.527153 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.527157 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.527162 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.527166 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.527170 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.527174 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.527179 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.527183 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.527187 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.527191 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.527202 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.527207 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.527211 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.527216 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.527220 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.527224 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.527229 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.527233 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.527237 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.527241 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.527246 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.527250 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.527254 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.527259 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.527263 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.527267 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.527271 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.527276 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.527280 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.527284 | controller |       09:41:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.527288 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.527293 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.527297 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.527303 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.527307 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.527312 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.527316 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.527320 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.527324 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.527329 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.527333 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.527337 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.527341 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.527348 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.527361 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.527366 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.527370 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.527374 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.527379 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.527383 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.527389 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.527398 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.527403 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.527407 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.527412 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.527416 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.527420 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.527424 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.527429 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.527433 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.527440 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.527444 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.527449 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.527453 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.527457 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.527462 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.527466 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.527470 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.527476 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.527481 | controller |       09:41:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.527485 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.527489 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.527494 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.527498 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.527504 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.527508 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.527513 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.527517 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.527521 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.527525 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.527530 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.527534 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.527538 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.527543 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.527547 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.527551 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.527555 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.527560 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.527564 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.527568 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.527573 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.527577 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.527581 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.527593 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.527610 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.527615 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.527619 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.527624 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.527628 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.527632 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.527636 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.527641 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.527645 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.527649 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.527654 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.527658 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.527662 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.527668 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.527673 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.527677 | controller |       09:41:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.527681 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.527686 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.527690 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.527694 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.527698 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.527704 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.527709 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.527715 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.527721 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.527727 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.527731 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.527735 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.527740 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.527744 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.527748 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.527753 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.527757 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.527761 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.527765 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.527770 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.527774 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.527778 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.527782 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.527787 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.527797 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.527804 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.527808 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.527813 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.527817 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.527823 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.527827 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.527832 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.527836 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.527840 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.527845 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.527849 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.527853 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.527857 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.527862 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.527866 | controller |       09:41:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.527870 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.527875 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.527880 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.527885 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.527890 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.527895 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.527899 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.527903 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.527908 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.527912 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.527916 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.527920 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.527925 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.527929 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.527933 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.527937 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.527942 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.527946 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.527950 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.527954 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.527960 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.527965 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.527969 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.527973 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.527977 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.527982 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.527991 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.527996 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.528000 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.528004 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.528009 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.528013 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.528017 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.528021 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.528026 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.528030 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.528034 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.528038 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.528043 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.528047 | controller |       09:41:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.528051 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.528055 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.528060 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.528064 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.528068 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.528073 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.528077 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.528081 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.528085 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.528090 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.528094 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.528098 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.528102 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.528108 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.528113 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.528117 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.528121 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.528126 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.528130 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.528134 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.528138 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.528144 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.528148 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.528153 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.528157 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.528161 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.528165 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.528175 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.528179 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.528183 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.528188 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.528192 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.528196 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.528200 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.528205 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.528209 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.528213 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.528217 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.528222 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.528226 | controller |       09:41:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.528230 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.528234 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.528239 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.528243 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.528249 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.528253 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.528257 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.528262 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.528266 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.528270 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.528275 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.528279 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.528283 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.528287 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.528292 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.528297 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.528302 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.528306 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.528310 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.528315 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.528320 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.528325 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.528329 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.528333 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.528337 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.528342 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.528346 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.528350 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.528354 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.528364 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.528368 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.528372 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.528377 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.528381 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.528386 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.528390 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.528394 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.528402 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.528406 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.528411 | controller |       09:41:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.528415 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.528421 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.528425 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.528429 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.528434 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.528438 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.528442 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.528446 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.528451 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.528455 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.528459 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.528464 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.528468 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.528472 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.528476 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.528481 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.528485 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.528489 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.528493 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.528498 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.528502 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.528506 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.528510 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.528515 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.528519 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.528523 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.528527 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.528532 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.528536 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.528542 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.528551 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.528556 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.528560 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.528565 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.528569 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.528573 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.528577 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.528582 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.528586 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.528590 | controller |       09:41:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.528608 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.528614 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.528619 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.528623 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.528627 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.528631 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.528636 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.528640 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.528644 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.528648 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.528653 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.528657 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.528661 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.528665 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.528670 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.528674 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.528680 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.528684 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.528689 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.528693 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.528699 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.528703 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.528708 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.528712 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.528716 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.528720 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.528725 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.528729 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.528733 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.528737 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.528743 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.528753 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.528758 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.528762 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.528766 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.528771 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.528777 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.528783 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.528789 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.528795 | controller |       09:41:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.528801 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.528806 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.528812 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.528818 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.528824 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.528828 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.528833 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.528837 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.528841 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.528845 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.528850 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.528854 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.528858 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.528865 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.528870 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.528874 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.528878 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.528882 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.528887 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.528891 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.528895 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.528899 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.528904 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.528908 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.528912 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.528916 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.528921 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.528925 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.528929 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.528933 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.528938 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.528942 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.528946 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.528957 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.528961 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.528966 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.528970 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.528974 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.528979 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.528983 | controller |       09:41:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.528990 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.528994 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.528998 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.529003 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.529009 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.529013 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.529017 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.529022 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.529026 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.529030 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.529034 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.529039 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.529043 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.529047 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.529052 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.529056 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.529060 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.529064 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.529069 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.529073 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.529077 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.529082 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.529086 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.529090 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.529094 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.529099 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.529103 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.529107 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.529112 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.529116 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.529120 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.529124 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.529129 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.529133 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.529143 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.529147 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.529152 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.529158 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.529162 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.529166 | controller |       09:41:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.529171 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.529175 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.529179 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.529184 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.529189 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.529194 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.529198 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.529202 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.529206 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.529211 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.529215 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.529219 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.529223 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.529228 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.529234 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.529238 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.529242 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.529246 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.529251 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.529255 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.529261 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.529266 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.529270 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.529275 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.529279 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.529283 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.529288 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.529292 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.529296 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.529302 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.529307 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.529311 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.529315 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.529319 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.529324 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.529333 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.529338 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.529342 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.529346 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.529351 | controller |       09:41:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.529355 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.529359 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.529363 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.529368 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.529372 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.529376 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.529381 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.529385 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.529389 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.529393 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.529398 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.529402 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.529406 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.529410 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.529415 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.529419 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.529423 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.529428 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.529432 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.529436 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.529442 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.529446 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.529451 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.529455 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.529459 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.529463 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.529468 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.529472 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.529476 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.529480 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.529485 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.529489 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.529493 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.529497 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.529502 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.529506 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.529519 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.529523 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.529528 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.529532 | controller |       09:41:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.529536 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.529541 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.529545 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.529549 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.529553 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.529559 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.529564 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.529569 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.529574 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.529578 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.529582 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.529587 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.529591 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.529608 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.529614 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.529619 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.529625 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.529629 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.529633 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.529638 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.529642 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.529646 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.529650 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.529655 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.529659 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.529663 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.529667 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.529671 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.529676 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.529680 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.529684 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.529689 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.529693 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.529697 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.529701 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.529706 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.529710 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.529714 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.529724 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.529729 | controller |       09:41:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.529733 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.529738 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.529742 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.529746 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.529752 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.529757 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.529761 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.529765 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.529770 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.529774 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.529778 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.529782 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.529787 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.529793 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.529797 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.529802 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.529806 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.529810 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.529814 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.529819 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.529823 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.529827 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.529831 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.529836 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.529840 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.529844 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.529848 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.529853 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.529857 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.529861 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.529865 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.529870 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.529874 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.529878 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.529882 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.529887 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.529891 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.529897 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.529903 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.529916 | controller |       09:41:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.529922 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.529926 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.529931 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.529935 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.529939 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.529943 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.529947 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.529952 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.529956 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.529960 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.529965 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.529969 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.529973 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.529977 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.529982 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.529986 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.529990 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.529994 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.529999 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.530003 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.530007 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.530011 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.530016 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.530020 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.530024 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.530028 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.530033 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.530037 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.530041 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.530047 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.530054 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.530058 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.530062 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.530067 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.530071 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.530077 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.530081 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.530085 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.530089 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.530094 | controller |       09:41:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.530104 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.530108 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.530113 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.530117 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.530121 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.530125 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.530130 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.530134 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.530138 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.530142 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.530147 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.530151 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.530155 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.530159 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.530164 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.530169 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.530175 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.530181 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.530186 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.530191 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.530199 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.530204 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.530209 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.530213 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.530217 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.530221 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.530226 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.530230 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.530234 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.530238 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.530243 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.530247 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.530251 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.530255 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.530260 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.530264 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.530268 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.530272 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.530277 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.530281 | controller |       09:41:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.530285 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.530289 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.530299 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.530304 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.530308 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.530313 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.530317 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.530321 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.530325 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.530330 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.530334 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.530338 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.530344 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.530350 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.530355 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.530359 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.530363 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.530367 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.530372 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.530376 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.530380 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.530384 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.530389 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.530393 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.530397 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.530402 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.530406 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.530410 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.530414 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.530419 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.530423 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.530427 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.530431 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.530436 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.530440 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.530444 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.530449 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.530453 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.530457 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.530461 | controller |       09:41:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.530466 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.530470 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.530474 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.530484 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.530493 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.530497 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.530502 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.530506 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.530510 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.530514 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.530519 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.530523 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.530527 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.530531 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.530535 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.530540 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.530544 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.530548 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.530553 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.530557 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.530561 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.530565 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.530570 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.530574 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.530578 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.530582 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.530587 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.530591 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.530605 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.530612 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.530616 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.530620 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.530626 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.530631 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.530635 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.530639 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.530644 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.530650 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.530654 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.530658 | controller |       09:41:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.530663 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.530667 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.530671 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.530675 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.530680 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.530689 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.530694 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.530698 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.530703 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.530707 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.530711 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.530716 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.530720 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.530724 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.530728 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.530733 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.530737 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.530741 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.530745 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.530750 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.530754 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.530758 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.530763 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.530767 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.530771 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.530775 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.530780 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.530784 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.530788 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.530794 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.530798 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.530803 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.530807 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.530811 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.530816 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.530820 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.530824 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.530828 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.530832 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.530837 | controller |       09:41:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.530841 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.530845 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.530849 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.530854 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.530858 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.530862 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.530872 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.530876 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.530880 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.530885 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.530889 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.530910 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.530915 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.530919 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.530924 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.530929 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.530934 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.530938 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.530943 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.530947 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.530953 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.530957 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.530961 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.530966 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.530970 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.530974 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.530978 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.530983 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.530987 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.530991 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.530995 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.531000 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.531004 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.531008 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.531012 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.531017 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.531021 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.531025 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.531029 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.531034 | controller |       09:41:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.531038 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.531042 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.531046 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.531051 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.531055 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.531059 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.531063 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.531073 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.531078 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.531082 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.531086 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.531090 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.531095 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.531101 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.531105 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.531109 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.531113 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.531117 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.531122 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.531126 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.531132 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.531136 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.531140 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.531145 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.531149 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.531153 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.531157 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.531162 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.531167 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.531172 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.531176 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.531180 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.531184 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.531189 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.531193 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.531197 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.531201 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.531206 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.531210 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.531214 | controller |       09:41:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.531219 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.531268 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.531273 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.531278 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.531284 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.531289 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.531293 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.531297 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.531301 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.531313 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.531318 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.531322 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.531326 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.531330 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.531335 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.531339 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.531343 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.531347 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.531352 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.531356 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.531360 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.531365 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.531369 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.531373 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.531377 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.531382 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.531386 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.531390 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.531395 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.531399 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.531405 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.531409 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.531414 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.531418 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.531422 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.531427 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.531431 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.531437 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.531441 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.531445 | controller |       09:41:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.531450 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.531454 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.531458 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.531462 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.531467 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.531471 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.531475 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.531479 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.531483 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.531488 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.531500 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.531505 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.531509 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.531513 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.531518 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.531522 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.531526 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.531531 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.531535 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.531539 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.531543 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.531548 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.531552 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.531556 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.531560 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.531565 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.531569 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.531573 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.531577 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.531585 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.531589 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.531593 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.531611 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.531615 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.531619 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.531623 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.531628 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.531632 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.531636 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.531641 | controller |       09:41:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.531645 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.531649 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.531654 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.531658 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.531662 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.531666 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.531671 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.531675 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.531679 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.531683 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.531688 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.531692 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.531703 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.531708 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.531712 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.531716 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.531721 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.531725 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.531729 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.531734 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.531740 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.531744 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.531749 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.531753 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.531757 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.531761 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.531766 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.531772 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.531776 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.531780 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.531784 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.531789 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.531793 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.531797 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.531801 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.531806 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.531810 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.531814 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.531820 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.531824 | controller |       09:41:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.531828 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.531833 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.531837 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.531841 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.531847 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.531851 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.531855 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.531860 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.531864 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.531868 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.531872 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.531877 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.531881 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.531893 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.531898 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.531902 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.531906 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.531910 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.531915 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.531919 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.531923 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.531928 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.531932 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.531936 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.531941 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.531945 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.531949 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.531953 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.531958 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.531962 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.531966 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.531971 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.531975 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.531979 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.531984 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.531988 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.531992 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.531996 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.532001 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.532005 | controller |       09:41:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.532009 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.532014 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.532018 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.532022 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.532028 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.532032 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.532037 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.532041 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.532045 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.532051 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.532056 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.532060 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.532064 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.532068 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.532072 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.532085 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.532092 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.532097 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.532103 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.532109 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.532118 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.532123 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.532129 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.532135 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.532140 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.532145 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.532149 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.532153 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.532158 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.532162 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.532166 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.532170 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.532175 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.532179 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.532183 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.532188 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.532192 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.532198 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.532202 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.532207 | controller |       09:41:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.532211 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.532215 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.532219 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.532224 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.532228 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.532232 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.532236 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.532241 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.532245 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.532249 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.532254 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.532258 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.532262 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.532266 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.532271 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.532277 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.532288 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.532293 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.532297 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.532301 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.532306 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.532310 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.532314 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.532319 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.532323 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.532329 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.532333 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.532337 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.532342 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.532348 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.532352 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.532356 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.532360 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.532365 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.532369 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.532373 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.532378 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.532382 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.532386 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.532390 | controller |       09:41:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.532395 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.532399 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.532403 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.532407 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.532412 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.532416 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.532420 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.532424 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.532429 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.532433 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.532437 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.532443 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.532448 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.532454 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.532459 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.532464 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.532470 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.532484 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.532490 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.532496 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.532504 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.532510 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.532515 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.532521 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.532525 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.532530 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.532534 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.532538 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.532542 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.532547 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.532551 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.532555 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.532559 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.532564 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.532568 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.532572 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.532675 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.532682 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.532686 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.532690 | controller |       09:41:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.532695 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.532699 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.532703 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.532708 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.532712 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.532716 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.532720 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.532725 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.532732 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.532736 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.532740 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.532745 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.532749 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.532757 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.532761 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.532765 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.532770 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.532774 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.532786 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.532790 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.532795 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.532799 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.532803 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.532808 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.532812 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.532816 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.532821 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.532825 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.532829 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.532833 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.532839 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.532844 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.532848 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.532852 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.532857 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.532861 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.532865 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.532869 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.532874 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.532878 | controller |       09:41:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.532882 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.532888 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.532893 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.532897 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.532903 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.532907 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.532911 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.532916 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.532920 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.532924 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.532928 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.532933 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.532937 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.532941 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.532945 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.532950 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.532954 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.532958 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.532962 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.532967 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.532976 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.532981 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.532985 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.532990 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.532996 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.533000 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.533004 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.533008 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.533013 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.533017 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.533021 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.533025 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.533030 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.533034 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.533038 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.533042 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.533047 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.533053 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.533057 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.533061 | controller |       09:41:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.533065 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.533070 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.533074 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.533078 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.533082 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.533087 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.533091 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.533095 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.533099 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.533104 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.533108 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.533112 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.533116 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.533121 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.533125 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.533129 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.533133 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.533138 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.533142 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.533146 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.533152 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.533162 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.533166 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.533171 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.533175 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.533180 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.533184 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.533189 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.533195 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.533203 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.533210 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.533214 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.533219 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.533223 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.533227 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.533232 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.533236 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.533241 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.533245 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.533249 | controller |       09:41:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.533254 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.533258 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.533262 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.533269 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.533274 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.533279 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.533283 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.533288 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.533292 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.533296 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.533301 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.533305 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.533309 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.533314 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.533318 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.533322 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.533326 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.533331 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.533335 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.533340 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.533346 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.533350 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.533361 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.533366 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.533370 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.533374 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.533379 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.533383 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.533387 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.533392 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.533396 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.533400 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.533405 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.533409 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.533413 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.533418 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.533422 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.533426 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.533431 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.533435 | controller |       09:41:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.533439 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.533444 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.533448 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.533452 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.533457 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.533461 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.533465 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.533470 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.533474 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.533478 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.533483 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.533487 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.533491 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.533497 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.533502 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.533506 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.533510 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.533515 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.533519 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.533523 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.533528 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.533532 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.533536 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.533542 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.533552 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.533557 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.533561 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.533565 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.533570 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.533574 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.533578 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.533583 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.533587 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.533591 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.533610 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.533615 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.533620 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.533624 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.533629 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.533633 | controller |       09:41:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.533637 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:41:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.533642 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.533646 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.533650 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.533657 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.533661 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.533665 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.533670 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.533674 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.533678 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.533683 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.533687 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.533691 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.533696 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.533700 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.533706 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.533711 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.533715 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.533719 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.533724 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.533728 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.533733 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.533737 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.533741 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.533746 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.533756 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.533760 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.533765 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.533769 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.533774 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.533778 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.533782 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.533787 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.533791 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.533795 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.533800 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.533804 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.533812 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:41:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.533816 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.533820 | controller |       09:41:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.533825 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.533829 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.533836 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.533840 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.533844 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.533849 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.533853 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.533857 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.533862 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.533866 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.533870 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.533875 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.533879 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.533883 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.533888 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.533892 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.533896 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.533901 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.533905 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.533909 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.533914 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.533918 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.533922 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.533927 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.533931 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.533935 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.533940 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.533949 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.533954 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.533961 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.533965 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.533969 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.533974 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.533978 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.533982 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.533987 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.533991 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.533996 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.534000 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.534004 | controller |       09:42:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.534009 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.534013 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.534017 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.534022 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.534026 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.534032 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.534037 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.534041 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.534045 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.534050 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.534054 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.534058 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.534063 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.534067 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.534071 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.534075 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.534080 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.534084 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.534088 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.534093 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.534099 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.534104 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.534109 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.534113 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.534117 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.534122 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.534126 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.534132 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.534141 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.534146 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.534151 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.534155 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.534161 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.534165 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.534170 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.534174 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.534178 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.534182 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.534187 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.534191 | controller |       09:42:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.534195 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.534200 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.534204 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.534208 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.534212 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.534217 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.534221 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.534225 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.534230 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.534234 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.534238 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.534242 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.534247 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.534253 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.534257 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.534261 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.534265 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.534270 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.534274 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.534278 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.534283 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.534287 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.534291 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.534295 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.534300 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.534304 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.534308 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.534312 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.534317 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.534321 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.534331 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.534335 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.534340 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.534344 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.534348 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.534352 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.534357 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.534361 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.534365 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.534370 | controller |       09:42:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.534374 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.534380 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.534384 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.534388 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.534395 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.534399 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.534403 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.534407 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.534412 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.534416 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.534420 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.534424 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.534429 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.534433 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.534437 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.534441 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.534446 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.534450 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.534454 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.534458 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.534463 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.534467 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.534471 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.534475 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.534480 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.534484 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.534488 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.534492 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.534497 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.534501 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.534505 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.534516 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.534521 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.534525 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.534530 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.534534 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.534538 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.534544 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.534549 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.534553 | controller |       09:42:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.534557 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.534561 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.534566 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.534570 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.534576 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.534580 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.534585 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.534589 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.534604 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.534610 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.534614 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.534628 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.534632 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.534637 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.534641 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.534645 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.534651 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.534656 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.534660 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.534664 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.534668 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.534673 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.534677 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.534681 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.534685 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.534689 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.534694 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.534698 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.534702 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.534710 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.534714 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.534718 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.534729 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.534734 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.534738 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.534742 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.534747 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.534751 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.534755 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.534760 | controller |       09:42:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.534764 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.534768 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.534773 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.534777 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.534781 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.534785 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.534790 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.534794 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.534798 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.534802 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.534807 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.534811 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.534815 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.534819 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.534824 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.534828 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.534832 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.534837 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.534841 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.534845 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.534851 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.534856 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.534860 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.534864 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.534868 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.534873 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.534877 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.534881 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.534885 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.534890 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.534907 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.534913 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.534918 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.534928 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.534932 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.534937 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.534941 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.534945 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.534949 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.534954 | controller |       09:42:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.534961 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.534965 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.534969 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.534974 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.534978 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.534982 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.534986 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.534991 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.534995 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.534999 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.535003 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.535008 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.535012 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.535018 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.535022 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.535028 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.535032 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.535037 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.535041 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.535045 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.535051 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.535055 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.535060 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.535064 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.535068 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.535072 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.535077 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.535081 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.535085 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.535089 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.535094 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.535098 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.535102 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.535106 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.535111 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.535120 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.535125 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.535129 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.535134 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.535138 | controller |       09:42:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.535142 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.535147 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.535151 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.535155 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.535161 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.535165 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.535170 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.535174 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.535178 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.535182 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.535187 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.535191 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.535195 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.535199 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.535205 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.535210 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.535214 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.535218 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.535222 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.535227 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.535231 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.535235 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.535239 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.535244 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.535248 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.535252 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.535256 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.535261 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.535265 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.535269 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.535273 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.535278 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.535282 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.535286 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.535291 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.535295 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.535304 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.535311 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.535315 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.535319 | controller |       09:42:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.535324 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.535331 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.535337 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.535343 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.535349 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.535355 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.535361 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.535367 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.535372 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.535376 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.535380 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.535384 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.535389 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.535393 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.535397 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.535401 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.535406 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.535410 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.535414 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.535418 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.535423 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.535427 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.535431 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.535435 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.535440 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.535444 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.535448 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.535452 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.535457 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.535463 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.535470 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.535474 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.535478 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.535483 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.535487 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.535491 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.535495 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.535500 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.535512 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.535517 | controller |       09:42:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.535522 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.535526 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.535530 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.535534 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.535539 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.535543 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.535547 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.535551 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.535556 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.535560 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.535564 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.535569 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.535573 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.535577 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.535581 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.535586 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.535590 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.535605 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.535612 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.535616 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.535622 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.535627 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.535631 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.535635 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.535639 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.535644 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.535648 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.535652 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.535656 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.535661 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.535665 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.535669 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.535673 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.535678 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.535682 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.535686 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.535690 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.535695 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.535699 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.535709 | controller |       09:42:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.535714 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.535718 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.535722 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.535726 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.535731 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.535735 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.535739 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.535743 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.535748 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.535752 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.535756 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.535760 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.535765 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.535773 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.535777 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.535781 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.535785 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.535790 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.535794 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.535798 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.535802 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.535807 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.535811 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.535815 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.535820 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.535824 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.535828 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.535832 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.535837 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.535841 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.535845 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.535849 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.535853 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.535858 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.535862 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.535866 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.535871 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.535875 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.535879 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.535883 | controller |       09:42:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.535893 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.535898 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.535902 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.535906 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.535916 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.535920 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.535925 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.535929 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.535933 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.535937 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.535941 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.535946 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.535950 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.535954 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.535958 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.535963 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.535967 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.535971 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.535975 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.535980 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.535985 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.535990 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.535994 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.535998 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.536002 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.536007 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.536011 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.536015 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.536019 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.536024 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.536028 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.536032 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.536036 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.536041 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.536045 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.536051 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.536055 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.536061 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.536065 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.536070 | controller |       09:42:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.536074 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.536078 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.536088 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.536092 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.536097 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.536101 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.536106 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.536110 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.536114 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.536118 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.536123 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.536127 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.536131 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.536135 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.536140 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.536144 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.536148 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.536152 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.536157 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.536161 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.536165 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.536170 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.536174 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.536178 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.536182 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.536187 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.536191 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.536195 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.536199 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.536206 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.536210 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.536214 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.536218 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.536223 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.536227 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.536231 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.536235 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.536240 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.536246 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.536250 | controller |       09:42:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.536255 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.536259 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.536263 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.536273 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.536277 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.536281 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.536285 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.536290 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.536294 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.536298 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.536303 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.536307 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.536313 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.536317 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.536321 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.536327 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.536331 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.536336 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.536340 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.536344 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.536350 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.536355 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.536359 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.536363 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.536367 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.536372 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.536376 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.536380 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.536384 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.536388 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.536393 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.536397 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.536401 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.536406 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.536410 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.536414 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.536418 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.536423 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.536427 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.536431 | controller |       09:42:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.536436 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.536440 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.536444 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.536448 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.536458 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.536462 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.536466 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.536471 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.536475 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.536479 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.536484 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.536488 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.536492 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.536498 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.536503 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.536507 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.536511 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.536515 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.536520 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.536524 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.536528 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.536532 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.536537 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.536541 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.536545 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.536549 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.536554 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.536558 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.536562 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.536566 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.536571 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.536575 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.536581 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.536585 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.536590 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.536611 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.536617 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.536622 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.536626 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.536630 | controller |       09:42:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.536634 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.536639 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.536643 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.536647 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.536654 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.536659 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.536669 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.536676 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.536682 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.536688 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.536693 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.536697 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.536701 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.536706 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.536710 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.536714 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.536718 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.536723 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.536727 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.536731 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.536736 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.536740 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.536744 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.536749 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.536753 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.536757 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.536761 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.536766 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.536770 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.536774 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.536781 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.536785 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.536790 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.536794 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.536800 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.536806 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.536812 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.536818 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.536822 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.536827 | controller |       09:42:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.536831 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.536835 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.536840 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.536844 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.536848 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.536853 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.536857 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.536868 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.536873 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.536877 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.536881 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.536889 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.536894 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.536898 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.536902 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.536906 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.536911 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.536915 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.536919 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.536924 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.536929 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.536934 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.536938 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.536942 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.536946 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.536951 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.536955 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.536959 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.536963 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.536969 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.536974 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.536978 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.536982 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.536987 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.536991 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.536995 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.537000 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.537004 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.537008 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.537012 | controller |       09:42:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.537018 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.537024 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.537029 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.537035 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.537039 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.537044 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.537048 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.537052 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.537057 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.537067 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.537072 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.537076 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.537081 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.537085 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.537090 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.537134 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.537144 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.537150 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.537156 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.537161 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.537201 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.537209 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.537215 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.537221 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.537226 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.537232 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.537236 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.537240 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.537247 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.537251 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.537256 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.537260 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.537264 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.537268 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.537273 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.537277 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.537281 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.537286 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.537290 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.537294 | controller |       09:42:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.537299 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.537303 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.537307 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.537311 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.537317 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.537321 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.537326 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.537330 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.537334 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.537339 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.537351 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.537356 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.537360 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.537367 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.537371 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.537375 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.537380 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.537384 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.537388 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.537393 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.537397 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.537402 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.537406 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.537410 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.537414 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.537419 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.537423 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.537427 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.537432 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.537436 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.537440 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.537445 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.537449 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.537454 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.537459 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.537463 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.537467 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.537472 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.537476 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.537481 | controller |       09:42:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.537485 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.537489 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.537494 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.537498 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.537504 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.537508 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.537513 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.537517 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.537521 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.537526 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.537532 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.537536 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.537546 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.537551 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.537555 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.537560 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.537564 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.537568 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.537573 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.537577 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.537581 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.537586 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.537590 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.537605 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.537612 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.537625 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.537629 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.537634 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.537638 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.537643 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.537647 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.537651 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.537656 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.537660 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.537665 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.537669 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.537673 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.537680 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.537684 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.537688 | controller |       09:42:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.537693 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.537697 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.537701 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.537706 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.537710 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.537715 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.537719 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.537723 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.537728 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.537732 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.537736 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.537741 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.537745 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.537756 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.537761 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.537767 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.537772 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.537776 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.537780 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.537785 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.537790 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.537795 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.537799 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.537803 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.537808 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.537812 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.537816 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.537822 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.537827 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.537833 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.537837 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.537842 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.537846 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.537850 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.537855 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.537860 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.537866 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.537872 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.537878 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.537885 | controller |       09:42:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.537891 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.537897 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.537903 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.537907 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.537912 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.537916 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.537920 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.537925 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.537929 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.537933 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.537938 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.537942 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.537946 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.537951 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.537961 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.537966 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.537971 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.537975 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.537979 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.537984 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.537992 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.537997 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.538001 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.538006 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.538010 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.538014 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.538019 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.538023 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.538028 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.538032 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.538036 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.538041 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.538045 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.538049 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.538054 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.538058 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.538063 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.538067 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.538071 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.538076 | controller |       09:42:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.538080 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.538084 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.538089 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.538093 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.538097 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.538102 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.538106 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.538110 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.538115 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.538122 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.538126 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.538131 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.538135 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.538141 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.538146 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.538150 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.538160 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.538165 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.538169 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.538174 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.538178 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.538182 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.538187 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.538191 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.538195 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.538200 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.538204 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.538208 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.538213 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.538217 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.538223 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.538227 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.538232 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.538236 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.538241 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.538245 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.538249 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.538253 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.538258 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.538262 | controller |       09:42:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.538267 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.538271 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.538275 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.538280 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.538286 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.538290 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.538294 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.538299 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.538303 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.538307 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.538312 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.538316 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.538320 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.538325 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.538329 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.538334 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.538338 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.538347 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.538352 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.538357 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.538361 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.538365 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.538370 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.538374 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.538378 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.538384 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.538389 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.538393 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.538398 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.538402 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.538406 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.538411 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.538415 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.538419 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.538424 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.538428 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.538432 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.538438 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.538443 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.538447 | controller |       09:42:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.538452 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.538456 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.538460 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.538465 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.538469 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.538475 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.538480 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.538484 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.538488 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.538493 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.538497 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.538501 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.538506 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.538510 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.538514 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.538519 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.538523 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.538527 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.538537 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.538542 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.538546 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.538551 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.538555 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.538559 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.538564 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.538568 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.538572 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.538577 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.538581 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.538589 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.538605 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.538612 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.538616 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.538621 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.538625 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.538630 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.538634 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.538638 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.538643 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.538647 | controller |       09:42:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.538651 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.538656 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.538660 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.538665 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.538671 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.538675 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.538680 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.538684 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.538690 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.538694 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.538699 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.538703 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.538707 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.538712 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.538716 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.538720 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.538725 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.538729 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.538733 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.538744 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.538750 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.538755 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.538759 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.538764 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.538768 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.538773 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.538777 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.538781 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.538786 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.538790 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.538794 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.538799 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.538803 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.538808 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.538812 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.538816 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.538821 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.538825 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.538829 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.538834 | controller |       09:42:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.538838 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.538842 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.538847 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.538851 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.538856 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.538860 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.538864 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.538869 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.538873 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.538877 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.538882 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.538886 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.538890 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.538913 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.538919 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.538923 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.538928 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.538932 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.538936 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.538941 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.538950 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.538965 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.538972 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.538979 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.538987 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.538994 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.538998 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.539003 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.539007 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.539011 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.539016 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.539020 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.539025 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.539029 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.539033 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.539038 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.539042 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.539046 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.539051 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.539055 | controller |       09:42:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.539060 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.539064 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.539068 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.539073 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.539080 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.539085 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.539089 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.539094 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.539098 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.539102 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.539107 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.539111 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.539115 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.539120 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.539124 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.539130 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.539135 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.539139 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.539143 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.539148 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.539152 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.539157 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.539169 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.539174 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.539178 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.539182 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.539187 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.539191 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.539196 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.539200 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.539204 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.539209 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.539213 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.539217 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.539222 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.539226 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.539230 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.539237 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.539241 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.539245 | controller |       09:42:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.539250 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.539254 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.539259 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.539265 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.539269 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.539273 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.539278 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.539282 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.539286 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.539291 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.539295 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.539299 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.539304 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.539308 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.539312 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.539317 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.539321 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.539325 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.539330 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.539334 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.539338 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.539343 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.539347 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.539351 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.539362 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.539367 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.539371 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.539376 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.539380 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.539386 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.539391 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.539395 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.539400 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.539404 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.539408 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.539413 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.539417 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.539421 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.539426 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.539430 | controller |       09:42:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.539435 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.539439 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.539443 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.539448 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.539452 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.539456 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.539461 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.539465 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.539469 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.539474 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.539478 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.539482 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.539487 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.539491 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.539495 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.539502 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.539506 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.539511 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.539515 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.539519 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.539525 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.539530 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.539534 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.539540 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.539544 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.539555 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.539560 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.539564 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.539569 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.539573 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.539579 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.539584 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.539588 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.539593 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.539611 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.539626 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.539633 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.539639 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.539645 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.539651 | controller |       09:42:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.539657 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.539661 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.539665 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.539670 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.539674 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.539679 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.539683 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.539687 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.539692 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.539696 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.539700 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.539739 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.539747 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.539755 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.539760 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.539764 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.539794 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.539799 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.539804 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.539808 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.539812 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.539817 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.539821 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.539828 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.539833 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.539839 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.539844 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.539858 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.539863 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.539867 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.539872 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.539876 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.539880 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.539885 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.539889 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.539925 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.539930 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.539935 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.539939 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.539944 | controller |       09:42:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.539972 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.539980 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.539994 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.539999 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.540006 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.540010 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.540015 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.540019 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.540023 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.540028 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.540032 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.540036 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.540042 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.540079 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.540086 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.540091 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.540095 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.540099 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.540104 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.540108 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.540115 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.540119 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.540124 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.540128 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.540132 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.540137 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.540141 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.540145 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.540158 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.540162 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.540167 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.540171 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.540176 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.540180 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.540184 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.540189 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.540193 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.540200 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.540204 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.540208 | controller |       09:42:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.540213 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.540217 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.540222 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.540226 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.540232 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.540237 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.540241 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.540245 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.540250 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.540254 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.540258 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.540263 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.540267 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.540271 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.540276 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.540280 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.540284 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.540289 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.540293 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.540297 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.540302 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.540309 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.540313 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.540318 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.540322 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.540326 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.540331 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.540335 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.540339 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.540353 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.540358 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.540362 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.540366 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.540371 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.540375 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.540380 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.540384 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.540388 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.540393 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.540397 | controller |       09:42:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.540401 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.540406 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.540410 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.540414 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.540418 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.540423 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.540427 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.540431 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.540436 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.540440 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.540444 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.540449 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.540453 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.540457 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.540462 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.540466 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.540470 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.540475 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.540479 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.540483 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.540489 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.540494 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.540498 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.540502 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.540507 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.540511 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.540515 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.540520 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.540524 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.540528 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.540533 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.540542 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.540547 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.540551 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.540556 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.540560 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.540564 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.540569 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.540573 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.540578 | controller |       09:42:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.540582 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.540589 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.540605 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.540620 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.540625 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.540629 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.540634 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.540638 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.540643 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.540647 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.540651 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.540656 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.540660 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.540667 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.540671 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.540677 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.540681 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.540686 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.540690 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.540694 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.540699 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.540703 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.540707 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.540711 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.540716 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.540720 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.540724 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.540729 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.540733 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.540737 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.540741 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.540746 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.540756 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.540761 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.540766 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.540770 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.540774 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.540779 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.540783 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.540787 | controller |       09:42:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.540792 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.540796 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.540800 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.540804 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.540811 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.540815 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.540819 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.540824 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.540828 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.540832 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.540836 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.540841 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.540845 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.540849 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.540854 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.540858 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.540864 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.540868 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.540873 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.540877 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.540881 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.540885 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.540890 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.540894 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.540898 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.540903 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.540907 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.540911 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.540915 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.540920 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.540924 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.540928 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.540933 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.540942 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.540946 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.540951 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.540955 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.540961 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.540967 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.540971 | controller |       09:42:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.540976 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.540980 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.540984 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.540988 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.540993 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.540997 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.541001 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.541006 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.541010 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.541014 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.541019 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.541023 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.541027 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.541031 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.541036 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.541040 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.541044 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.541048 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.541053 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.541057 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.541063 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.541067 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.541072 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.541076 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.541080 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.541084 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.541089 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.541093 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.541097 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.541104 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.541113 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.541119 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.541125 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.541131 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.541138 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.541149 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.541153 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.541158 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.541162 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.541166 | controller |       09:42:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.541173 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.541177 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.541181 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.541186 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.541190 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.541194 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.541198 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.541203 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.541207 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.541211 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.541215 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.541220 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.541224 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.541228 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.541233 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.541237 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.541241 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.541245 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.541250 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.541254 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.541260 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.541264 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.541269 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.541273 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.541277 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.541282 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.541286 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.541290 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.541294 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.541299 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.541303 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.541307 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.541312 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.541316 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.541320 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.541325 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.541334 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.541338 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.541343 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.541347 | controller |       09:42:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.541352 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.541356 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.541360 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.541364 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.541369 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.541373 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.541377 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.541381 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.541386 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.541390 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.541394 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.541398 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.541403 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.541411 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.541417 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.541421 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.541426 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.541430 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.541434 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.541439 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.541443 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.541447 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.541452 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.541456 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.541460 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.541464 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.541469 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.541473 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.541477 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.541482 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.541486 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.541490 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.541494 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.541499 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.541503 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.541507 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.541512 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.541523 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.541528 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.541532 | controller |       09:42:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.541537 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.541541 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.541545 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.541549 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.541556 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.541560 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.541564 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.541568 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.541573 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.541577 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.541581 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.541585 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.541590 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.541605 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.541612 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.541616 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.541621 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.541625 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.541629 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.541634 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.541638 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.541642 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.541646 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.541651 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.541655 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.541659 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.541663 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.541668 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.541672 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.541676 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.541681 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.541685 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.541689 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.541693 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.541698 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.541702 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.541706 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.541713 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.541719 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.541730 | controller |       09:42:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.541734 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.541739 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.541743 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.541747 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.541752 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.541756 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.541760 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.541764 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.541769 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.541773 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.541777 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.541782 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.541786 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.541790 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.541794 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.541799 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.541803 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.541807 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.541812 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.541816 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.541820 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.541824 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.541829 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.541833 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.541837 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.541842 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.541846 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.541850 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.541854 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.541861 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.541865 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.541869 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.541873 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.541878 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.541882 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.541886 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.541891 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.541895 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.541899 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.541903 | controller |       09:42:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.541913 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.541917 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.541922 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.541926 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.541930 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.541934 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.541939 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.541943 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.541947 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.541951 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.541956 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.541960 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.541964 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.541970 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.541975 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.541980 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.541985 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.541989 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.541993 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.541998 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.542006 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.542011 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.542015 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.542019 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.542024 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.542028 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.542032 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.542036 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.542041 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.542045 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.542049 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.542053 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.542058 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.542062 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.542066 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.542071 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.542075 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.542079 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.542084 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.542088 | controller |       09:42:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.542092 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.542102 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.542106 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.542111 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.542115 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.542119 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.542124 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.542130 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.542134 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.542138 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.542143 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.542147 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.542151 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.542157 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.542162 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.542166 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.542170 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.542175 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.542179 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.542183 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.542188 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.542192 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.542196 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.542200 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.542205 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.542211 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.542217 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.542223 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.542229 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.542235 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.542241 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.542246 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.542250 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.542254 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.542259 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.542265 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.542270 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.542274 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.542278 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.542283 | controller |       09:42:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.542287 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.542291 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.542295 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.542306 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.542313 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.542317 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.542322 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.542326 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.542330 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.542334 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.542339 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.542343 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.542347 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.542352 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.542356 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.542360 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.542364 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.542369 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.542373 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.542377 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.542382 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.542386 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.542390 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.542394 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.542399 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.542403 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.542407 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.542412 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.542416 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.542420 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.542428 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.542432 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.542436 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.542441 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.542445 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.542449 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.542453 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.542459 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.542464 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.542468 | controller |       09:42:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.542472 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.542477 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.542481 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.542485 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.542495 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.542500 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.542504 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.542509 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.542513 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.542517 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.542521 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.542526 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.542532 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.542536 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.542540 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.542545 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.542549 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.542553 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.542557 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.542562 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.542566 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.542570 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.542575 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.542579 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.542583 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.542588 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.542592 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.542608 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.542613 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.542619 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.542624 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.542637 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.542641 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.542646 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.542650 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.542654 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.542658 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.542663 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.542667 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.542671 | controller |       09:42:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.542676 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.542680 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.542684 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.542688 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.542693 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.542697 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.542707 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.542712 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.542716 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.542721 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.542725 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.542729 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.542734 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.542738 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.542742 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.542747 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.542751 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.542755 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.542760 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.542764 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.542770 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.542774 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.542779 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.542783 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.542787 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.542792 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.542796 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.542800 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.542805 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.542809 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.542813 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.542818 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.542824 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.542828 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.542833 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.542837 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.542841 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.542846 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.542850 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.542854 | controller |       09:42:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.542859 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.542863 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.542867 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.542871 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.542877 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.542882 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.542886 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.542895 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.542912 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.542918 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.542922 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.542927 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.542931 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.542938 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.542942 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.542946 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.542951 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.542955 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.542959 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.542964 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.542970 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.542974 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.542978 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.542983 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.542987 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.542991 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.542996 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.543000 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.543004 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.543008 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.543013 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.543017 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.543021 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.543026 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.543030 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.543034 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.543038 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.543043 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.543047 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.543051 | controller |       09:42:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.543056 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.543060 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.543064 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.543069 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.543075 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.543080 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.543085 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.543089 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.543099 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.543104 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.543108 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.543113 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.543117 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.543121 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.543125 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.543130 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.543136 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.543140 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.543144 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.543148 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.543153 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.543157 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.543161 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.543166 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.543170 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.543174 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.543178 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.543183 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.543187 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.543191 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.543195 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.543200 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.543204 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.543208 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.543213 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.543217 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.543221 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.543227 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.543232 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.543236 | controller |       09:42:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.543240 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.543245 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.543249 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.543253 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.543257 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.543262 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.543266 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.543270 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.543274 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.543279 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.543288 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.543293 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.543297 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.543301 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.543306 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.543311 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.543316 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.543320 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.543325 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.543329 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.543333 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.543337 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.543342 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.543346 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.543350 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.543355 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.543359 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.543363 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.543369 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.543376 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.543381 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.543385 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.543389 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.543393 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.543398 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.543402 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.543406 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.543411 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.543415 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.543419 | controller |       09:42:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.543424 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.543428 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.543432 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.543437 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.543441 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.543445 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.543449 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.543454 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.543458 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.543462 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.543467 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.543476 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.543481 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.543485 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.543489 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.543494 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.543498 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.543502 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.543506 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.543511 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.543517 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.543521 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.543525 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.543530 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.543534 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.543538 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.543542 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.543547 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.543551 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.543555 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.543560 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.543564 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.543568 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.543572 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.543577 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.543581 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.543585 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.543590 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.543594 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.543611 | controller |       09:42:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.543615 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:42:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.543619 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.543624 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.543628 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.543632 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.543636 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.543641 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.543645 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.543649 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.543653 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.543660 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.543664 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.543668 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.543680 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.543685 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.543689 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.543693 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.543697 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.543702 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.543706 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.543710 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.543715 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.543719 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.543723 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.543728 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.543732 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.543736 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.543741 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.543745 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.543749 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.543755 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.543759 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.543764 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.543768 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.543772 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.543777 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.543781 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.543785 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:42:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.543789 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.543794 | controller |       09:42:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.543798 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.543802 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.543807 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.543811 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.543817 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.543821 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.543826 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.543830 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.543834 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.543838 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.543843 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.543847 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.543851 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.543856 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.543865 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.543869 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.543873 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.543878 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.543882 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.543886 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.543892 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.543897 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.543901 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.543906 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.543910 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.543914 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.543918 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.543924 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.543928 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.543933 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.543937 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.543941 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.543945 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.543950 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.543954 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.543958 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.543962 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.543969 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.543973 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.543977 | controller |       09:43:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.543981 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.543986 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.543990 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.543994 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.543999 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.544003 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.544007 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.544012 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.544016 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.544020 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.544024 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.544029 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.544033 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.544037 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.544042 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.544046 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.544056 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.544060 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.544064 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.544069 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.544073 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.544077 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.544082 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.544086 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.544090 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.544094 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.544099 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.544103 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.544107 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.544113 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.544118 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.544122 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.544126 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.544131 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.544135 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.544139 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.544143 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.544148 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.544152 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.544156 | controller |       09:43:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.544161 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.544165 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.544169 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.544174 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.544180 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.544184 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.544188 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.544193 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.544197 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.544202 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.544207 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.544211 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.544215 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.544220 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.544224 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.544228 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.544233 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.544242 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.544246 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.544251 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.544257 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.544261 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.544265 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.544270 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.544274 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.544278 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.544282 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.544287 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.544291 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.544295 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.544300 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.544304 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.544308 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.544312 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.544317 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.544321 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.544325 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.544329 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.544334 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.544338 | controller |       09:43:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.544342 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.544347 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.544351 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.544355 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.544360 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.544364 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.544368 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.544374 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.544380 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.544386 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.544392 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.544398 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.544404 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.544412 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.544417 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.544421 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.544425 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.544429 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.544440 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.544445 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.544449 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.544453 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.544458 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.544462 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.544466 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.544473 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.544477 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.544481 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.544486 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.544490 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.544494 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.544498 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.544503 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.544507 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.544511 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.544516 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.544520 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.544524 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.544528 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.544533 | controller |       09:43:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.544537 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.544541 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.544546 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.544550 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.544557 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.544561 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.544566 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.544570 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.544574 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.544578 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.544583 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.544587 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.544591 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.544609 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.544613 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.544620 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.544625 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.544629 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.544634 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.544644 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.544649 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.544653 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.544657 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.544662 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.544666 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.544670 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.544674 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.544679 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.544683 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.544687 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.544692 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.544696 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.544700 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.544705 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.544709 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.544713 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.544718 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.544724 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.544728 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.544733 | controller |       09:43:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.544737 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.544741 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.544745 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.544750 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.544754 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.544758 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.544763 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.544767 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.544773 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.544778 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.544782 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.544786 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.544791 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.544795 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.544799 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.544803 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.544808 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.544812 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.544816 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.544821 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.544827 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.544836 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.544841 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.544845 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.544849 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.544854 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.544858 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.544862 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.544866 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.544872 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.544877 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.544881 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.544885 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.544890 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.544894 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.544898 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.544902 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.544907 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.544911 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.544915 | controller |       09:43:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.544920 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.544924 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.544928 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.544933 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.544937 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.544941 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.544945 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.544950 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.544954 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.544958 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.544962 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.544967 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.544971 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.544975 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.544979 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.544984 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.544988 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.544992 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.544997 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.545001 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.545007 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.545011 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.545020 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.545025 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.545031 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.545035 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.545040 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.545044 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.545048 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.545053 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.545058 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.545063 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.545067 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.545072 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.545076 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.545080 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.545084 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.545089 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.545093 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.545097 | controller |       09:43:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.545102 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.545106 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.545110 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.545115 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.545119 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.545137 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.545141 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.545146 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.545150 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.545154 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.545159 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.545163 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.545167 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.545173 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.545178 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.545182 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.545186 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.545191 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.545195 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.545199 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.545204 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.545208 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.545212 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.545216 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.545226 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.545231 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.545235 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.545239 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.545244 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.545248 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.545252 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.545257 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.545261 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.545265 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.545270 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.545274 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.545278 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.545283 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.545289 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.545293 | controller |       09:43:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.545298 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.545302 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.545306 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.545312 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.545318 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.545323 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.545327 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.545331 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.545336 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.545340 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.545344 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.545349 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.545353 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.545357 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.545361 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.545366 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.545370 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.545375 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.545379 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.545383 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.545387 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.545392 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.545396 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.545400 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.545405 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.545414 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.545418 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.545423 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.545427 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.545432 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.545436 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.545440 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.545444 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.545449 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.545454 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.545460 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.545466 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.545475 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.545481 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.545487 | controller |       09:43:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.545493 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.545497 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.545501 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.545505 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.545514 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.545518 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.545522 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.545527 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.545531 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.545535 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.545539 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.545544 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.545548 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.545552 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.545557 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.545561 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.545565 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.545569 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.545574 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.545578 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.545582 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.545586 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.545591 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.545609 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.545615 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.545620 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.545624 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.545635 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.545640 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.545646 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.545651 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.545655 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.545659 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.545664 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.545668 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.545672 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.545677 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.545681 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.545685 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.545690 | controller |       09:43:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.545694 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.545698 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.545702 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.545707 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.545711 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.545715 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.545720 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.545724 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.545728 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.545732 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.545737 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.545741 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.545745 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.545749 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.545754 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.545758 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.545762 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.545767 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.545771 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.545775 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.545783 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.545787 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.545792 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.545796 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.545800 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.545805 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.545809 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.545813 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.545822 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.545827 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.545831 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.545836 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.545840 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.545844 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.545849 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.545853 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.545857 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.545861 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.545866 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.545870 | controller |       09:43:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.545874 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.545879 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.545884 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.545889 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.545893 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.545897 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.545902 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.545906 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.545910 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.545915 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.545919 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.545923 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.545927 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.545933 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.545938 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.545944 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.545948 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.545952 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.545957 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.545961 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.545965 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.545970 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.545974 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.545978 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.545982 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.545987 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.545991 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.545995 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.545999 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.546009 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.546013 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.546018 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.546022 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.546026 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.546030 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.546035 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.546039 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.546043 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.546048 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.546052 | controller |       09:43:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.546056 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.546061 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.546065 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.546265 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.546272 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.546276 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.546280 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.546284 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.546289 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.546293 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.546297 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.546302 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.546306 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.546310 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.546315 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.546319 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.546323 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.546328 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.546332 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.546336 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.546340 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.546346 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.546351 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.546355 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.546359 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.546364 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.546368 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.546372 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.546377 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.546381 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.546385 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.546397 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.546401 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.546406 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.546410 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.546414 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.546418 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.546424 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.546429 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.546433 | controller |       09:43:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.546437 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.546442 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.546446 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.546450 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.546455 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.546459 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.546463 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.546467 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.546472 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.546476 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.546480 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.546485 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.546489 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.546493 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.546497 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.546502 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.546506 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.546510 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.546514 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.546519 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.546523 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.546527 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.546532 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.546537 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.546543 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.546549 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.546555 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.546560 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.546566 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.546575 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.546582 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.546586 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.546610 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.546617 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.546621 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.546626 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.546630 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.546634 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.546639 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.546643 | controller |       09:43:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.546647 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.546653 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.546658 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.546662 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.546666 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.546671 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.546675 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.546679 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.546683 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.546688 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.546692 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.546696 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.546700 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.546705 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.546709 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.546713 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.546717 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.546722 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.546726 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.546730 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.546736 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.546741 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.546745 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.546749 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.546754 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.546758 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.546762 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.546766 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.546771 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.546775 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.546779 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.546784 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.546788 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.546799 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.546803 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.546807 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.546812 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.546816 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.546820 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.546825 | controller |       09:43:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.546829 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.546833 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.546837 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.546842 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.546846 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.546850 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.546855 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.546859 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.546863 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.546867 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.546872 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.546876 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.546880 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.546888 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.546892 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.546897 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.546950 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.546955 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.546960 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.546964 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.546970 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.546975 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.546979 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.546983 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.546988 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.546992 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.546997 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.547001 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.547005 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.547009 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.547014 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.547018 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.547023 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.547027 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.547039 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.547043 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.547048 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.547052 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.547057 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.547061 | controller |       09:43:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.547065 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.547070 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.547074 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.547078 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.547086 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.547091 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.547095 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.547100 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.547104 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.547108 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.547113 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.547117 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.547121 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.547126 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.547130 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.547134 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.547139 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.547143 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.547147 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.547152 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.547156 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.547160 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.547165 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.547169 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.547173 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.547178 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.547182 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.547186 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.547191 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.547195 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.547199 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.547205 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.547210 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.547214 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.547219 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.547223 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.547234 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.547240 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.547244 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.547249 | controller |       09:43:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.547255 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.547260 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.547264 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.547268 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.547273 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.547277 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.547282 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.547286 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.547290 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.547295 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.547299 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.547303 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.547308 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.547312 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.547316 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.547320 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.547325 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.547329 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.547333 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.547338 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.547342 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.547346 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.547351 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.547355 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.547359 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.547364 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.547368 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.547372 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.547377 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.547383 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.547387 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.547392 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.547396 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.547401 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.547405 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.547409 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.547414 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.547424 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.547429 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.547433 | controller |       09:43:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.547438 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.547442 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.547446 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.547451 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.547455 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.547459 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.547464 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.547468 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.547472 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.547477 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.547481 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.547486 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.547490 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.547494 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.547500 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.547506 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.547510 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.547515 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.547519 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.547523 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.547529 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.547534 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.547538 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.547542 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.547547 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.547551 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.547556 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.547560 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.547564 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.547568 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.547573 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.547577 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.547581 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.547586 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.547590 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.547606 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.547613 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.547618 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.547623 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.547638 | controller |       09:43:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.547645 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.547652 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.547658 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.547664 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.547668 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.547675 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.547680 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.547684 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.547689 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.547693 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.547697 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.547702 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.547706 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.547713 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.547717 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.547721 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.547726 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.547730 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.547735 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.547739 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.547743 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.547747 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.547752 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.547756 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.547760 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.547765 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.547769 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.547773 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.547778 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.547782 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.547786 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.547791 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.547795 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.547799 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.547804 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.547808 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.547812 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.547817 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.547823 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.547827 | controller |       09:43:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.547839 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.547844 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.547848 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.547852 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.547860 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.547864 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.547868 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.547873 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.547877 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.547882 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.547886 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.547890 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.547895 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.547899 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.547903 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.547908 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.547912 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.547916 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.547921 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.547925 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.547931 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.547936 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.547940 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.547944 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.547949 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.547953 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.547957 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.547962 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.547966 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.547970 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.547976 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.547981 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.547985 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.547989 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.547994 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.547998 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.548002 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.548008 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.548013 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.548017 | controller |       09:43:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.548021 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.548032 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.548037 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.548041 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.548046 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.548050 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.548054 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.548059 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.548063 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.548068 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.548072 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.548076 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.548080 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.548086 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.548091 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.548095 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.548100 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.548104 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.548108 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.548113 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.548117 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.548121 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.548126 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.548130 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.548134 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.548139 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.548143 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.548147 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.548151 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.548157 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.548162 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.548166 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.548170 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.548175 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.548179 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.548183 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.548188 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.548192 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.548196 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.548201 | controller |       09:43:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.548205 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.548210 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.548214 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.548223 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.548228 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.548232 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.548237 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.548241 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.548246 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.548250 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.548254 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.548259 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.548263 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.548267 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.548272 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.548276 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.548280 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.548284 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.548289 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.548293 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.548299 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.548303 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.548308 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.548312 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.548316 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.548321 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.548325 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.548329 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.548334 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.548338 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.548342 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.548347 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.548351 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.548355 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.548360 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.548366 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.548370 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.548375 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.548379 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.548383 | controller |       09:43:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.548388 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.548392 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.548396 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.548401 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.548413 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.548418 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.548422 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.548426 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.548431 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.548435 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.548439 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.548444 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.548448 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.548454 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.548458 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.548463 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.548467 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.548472 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.548476 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.548480 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.548485 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.548489 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.548493 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.548498 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.548502 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.548506 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.548511 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.548515 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.548519 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.548524 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.548528 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.548532 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.548537 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.548541 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.548545 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.548550 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.548554 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.548559 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.548563 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.548567 | controller |       09:43:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.548572 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.548576 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.548580 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.548585 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.548591 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.548614 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.548619 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.548623 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.548628 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.548632 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.548636 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.548641 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.548647 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.548652 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.548656 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.548660 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.548665 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.548669 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.548673 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.548678 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.548682 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.548686 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.548691 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.548695 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.548699 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.548704 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.548708 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.548712 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.548716 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.548721 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.548725 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.548731 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.548738 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.548744 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.548750 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.548757 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.548763 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.548770 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.548775 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.548779 | controller |       09:43:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.548784 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.548788 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.548792 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.548797 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.548801 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.548805 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.548810 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.548820 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.548825 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.548829 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.548834 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.548838 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.548843 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.548847 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.548851 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.548858 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.548862 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.548866 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.548871 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.548875 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.548885 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.548889 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.548894 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.548898 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.548902 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.548907 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.548911 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.548915 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.548920 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.548927 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.548932 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.548936 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.548943 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.548947 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.548951 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.548956 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.548960 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.548964 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.548969 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.548973 | controller |       09:43:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.548978 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.548982 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.548986 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.548991 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.548995 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.548999 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.549004 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.549008 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.549019 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.549024 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.549028 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.549033 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.549037 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.549041 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.549046 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.549050 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.549054 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.549059 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.549063 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.549067 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.549074 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.549078 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.549083 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.549087 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.549091 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.549096 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.549100 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.549104 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.549109 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.549113 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.549117 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.549122 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.549126 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.549130 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.549135 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.549139 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.549143 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.549148 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.549152 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.549156 | controller |       09:43:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.549161 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.549165 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.549169 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.549174 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.549178 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.549182 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.549187 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.549191 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.549195 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.549200 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.549209 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.549216 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.549220 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.549226 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.549231 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.549235 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.549239 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.549244 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.549248 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.549252 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.549257 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.549261 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.549265 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.549270 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.549274 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.549279 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.549283 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.549287 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.549292 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.549296 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.549302 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.549306 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.549311 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.549315 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.549320 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.549324 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.549328 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.549333 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.549337 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.549341 | controller |       09:43:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.549346 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.549350 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.549354 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.549359 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.549365 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.549369 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.549374 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.549378 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.549382 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.549387 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.549391 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.549401 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.549406 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.549410 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.549415 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.549419 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.549423 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.549428 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.549432 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.549436 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.549441 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.549445 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.549449 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.549454 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.549458 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.549462 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.549467 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.549471 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.549477 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.549482 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.549486 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.549490 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.549495 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.549499 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.549503 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.549508 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.549512 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.549518 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.549522 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.549527 | controller |       09:43:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.549531 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.549536 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.549540 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.549544 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.549549 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.549553 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.549557 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.549562 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.549566 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.549570 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.549575 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.549579 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.549583 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.549593 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.549611 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.549615 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.549620 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.549624 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.549629 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.549633 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.549637 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.549642 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.549646 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.549653 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.549658 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.549662 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.549666 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.549671 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.549675 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.549681 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.549686 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.549690 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.549694 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.549699 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.549703 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.549708 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.549712 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.549716 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.549721 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.549725 | controller |       09:43:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.549730 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.549734 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.549738 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.549742 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.549749 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.549753 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.549758 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.549762 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.549766 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.549771 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.549777 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.549781 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.549785 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.549790 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.549800 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.549805 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.549809 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.549813 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.549818 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.549822 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.549830 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.549834 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.549839 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.549843 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.549848 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.549852 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.549856 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.549861 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.549865 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.549869 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.549874 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.549878 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.549883 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.549887 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.549891 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.549896 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.549900 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.549905 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.549912 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.549917 | controller |       09:43:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.549921 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.549925 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.549930 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.549934 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.549938 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.549943 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.549947 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.549951 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.549956 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.549960 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.549964 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.549969 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.549973 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.549979 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.549983 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.549988 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.549998 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.550003 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.550007 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.550011 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.550016 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.550020 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.550024 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.550029 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.550033 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.550037 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.550042 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.550048 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.550053 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.550057 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.550061 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.550066 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.550070 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.550075 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.550079 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.550083 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.550088 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.550092 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.550096 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.550101 | controller |       09:43:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.550105 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.550109 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.550114 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.550118 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.550124 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.550128 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.550133 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.550137 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.550142 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.550146 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.550150 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.550154 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.550159 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.550163 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.550167 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.550281 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.550286 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.550297 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.550301 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.550306 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.550310 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.550315 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.550319 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.550323 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.550328 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.550332 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.550336 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.550341 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.550345 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.550350 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.550354 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.550358 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.550363 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.550367 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.550372 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.550376 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.550391 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.550397 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.550402 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.550406 | controller |       09:43:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.550410 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.550415 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.550419 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.550423 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.550428 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.550432 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.550436 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.550441 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.550445 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.550451 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.550456 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.550460 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.550465 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.550469 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.550473 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.550478 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.550482 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.550486 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.550497 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.550502 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.550506 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.550511 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.550515 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.550519 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.550524 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.550528 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.550532 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.550537 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.550541 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.550547 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.550552 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.550556 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.550560 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.550565 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.550569 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.550573 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.550578 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.550582 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.550586 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.550591 | controller |       09:43:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.550610 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.550616 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.550620 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.550625 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.550629 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.550634 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.550638 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.550642 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.550647 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.550651 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.550655 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.550659 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.550664 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.550668 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.550672 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.550677 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.550681 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.550685 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.550690 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.550701 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.550707 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.550712 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.550716 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.550720 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.550725 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.550731 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.550736 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.550740 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.550744 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.550749 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.550755 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.550759 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.550763 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.550768 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.550772 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.550777 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.550781 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.550785 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.550790 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.550794 | controller |       09:43:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.550798 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.550803 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.550807 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.550811 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.550816 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.550820 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.550824 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.550829 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.550833 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.550837 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.550842 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.550846 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.550850 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.550856 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.550861 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.550865 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.550869 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.550874 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.550878 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.550882 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.550896 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.550927 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.550932 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.550937 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.550941 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.550946 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.550950 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.550954 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.550959 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.550963 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.550967 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.550972 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.550976 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.550980 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.550985 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.550989 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.550993 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.550998 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.551002 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.551006 | controller |       09:43:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.551011 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.551015 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.551020 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.551024 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.551030 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.551035 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.551039 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.551044 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.551050 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.551054 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.551059 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.551063 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.551067 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.551072 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.551076 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.551080 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.551085 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.551089 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.551093 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.551098 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.551102 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.551107 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.551119 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.551124 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.551128 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.551132 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.551137 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.551141 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.551145 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.551150 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.551154 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.551159 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.551163 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.551167 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.551172 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.551176 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.551180 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.551188 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.551192 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.551197 | controller |       09:43:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.551201 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.551205 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.551210 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.551214 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.551220 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.551225 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.551229 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.551233 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.551238 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.551242 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.551246 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.551251 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.551255 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.551259 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.551264 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.551268 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.551272 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.551277 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.551281 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.551285 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.551290 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.551294 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.551298 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.551308 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.551315 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.551319 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.551324 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.551328 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.551332 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.551338 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.551343 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.551347 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.551351 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.551356 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.551360 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.551364 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.551369 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.551373 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.551377 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.551382 | controller |       09:43:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.551386 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.551390 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.551395 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.551399 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.551403 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.551417 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.551421 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.551426 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.551430 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.551434 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.551439 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.551443 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.551447 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.551452 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.551456 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.551460 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.551465 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.551469 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.551473 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.551478 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.551484 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.551488 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.551493 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.551497 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.551501 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.551512 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.551517 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.551521 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.551525 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.551530 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.551534 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.551539 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.551543 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.551548 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.551552 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.551556 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.551561 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.551565 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.551569 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.551574 | controller |       09:43:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.551578 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.551582 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.551587 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.551593 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.551611 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.551615 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.551620 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.551624 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.551628 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.551633 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.551637 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.551642 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.551647 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.551653 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.551658 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.551664 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.551668 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.551672 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.551677 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.551681 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.551686 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.551690 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.551694 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.551699 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.551703 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.551707 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.551718 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.551723 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.551727 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.551731 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.551736 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.551740 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.551745 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.551749 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.551754 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.551758 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.551762 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.551767 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.551771 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.551775 | controller |       09:43:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.551780 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.551784 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.551788 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.551793 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.551799 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.551803 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.551808 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.551812 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.551816 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.551821 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.551825 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.551829 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.551834 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.551839 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.551845 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.551851 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.551857 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.551863 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.551869 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.551875 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.551883 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.551887 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.551892 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.551897 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.551902 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.551906 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.551910 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.551914 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.551925 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.551930 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.551934 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.551938 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.551943 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.551947 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.551952 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.551956 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.551960 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.551967 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.551972 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.551978 | controller |       09:43:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.551983 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.551989 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.551995 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.552000 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.552006 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.552011 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.552016 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.552022 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.552027 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.552032 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.552038 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.552043 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.552048 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.552054 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.552059 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.552064 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.552069 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.552074 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.552080 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.552085 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.552090 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.552095 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.552101 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.552106 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.552111 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.552116 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.552122 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.552128 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.552133 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.552149 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.552157 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.552162 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.552168 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.552173 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.552179 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.552184 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.552189 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.552194 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.552200 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.552205 | controller |       09:43:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.552210 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.552216 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.552223 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.552228 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.552234 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.552243 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.552248 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.552254 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.552260 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.552266 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.552271 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.552277 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.552284 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.552290 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.552295 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.552301 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.552307 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.552313 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.552319 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.552324 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.552330 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.552335 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.552339 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.552344 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.552348 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.552352 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.552356 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.552361 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.552365 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.552369 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.552374 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.552386 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.552391 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.552395 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.552400 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.552404 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.552408 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.552412 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.552417 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.552421 | controller |       09:43:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.552425 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.552430 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.552434 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.552438 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.552443 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.552447 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.552451 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.552455 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.552460 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.552464 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.552468 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.552473 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.552477 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.552483 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.552487 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.552491 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.552496 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.552500 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.552504 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.552509 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.552513 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.552519 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.552524 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.552528 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.552532 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.552537 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.552541 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.552545 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.552549 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.552554 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.552558 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.552562 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.552573 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.552577 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.552582 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.552586 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.552590 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.552605 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.552612 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.552617 | controller |       09:43:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.552621 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.552625 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.552630 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.552634 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.552643 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.552648 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.552652 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.552656 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.552660 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.552665 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.552669 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.552673 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.552678 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.552682 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.552686 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.552690 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.552695 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.552699 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.552703 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.552707 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.552712 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.552716 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.552720 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.552725 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.552729 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.552733 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.552737 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.552742 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.552746 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.552752 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.552756 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.552761 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.552765 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.552776 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.552781 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.552785 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.552789 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.552796 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.552800 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.552804 | controller |       09:43:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.552809 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.552815 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.552819 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.552823 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.552828 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.552832 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.552836 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.552840 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.552845 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.552849 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.552853 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.552858 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.552862 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.552866 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.552870 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.552875 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.552879 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.552883 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.552887 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.552892 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.552898 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.552902 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.552906 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.552911 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.552915 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.552919 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.552923 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.552928 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.552932 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.552938 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.552942 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.552947 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.552951 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.552955 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.552966 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.552970 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.552975 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.552979 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.552983 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.552988 | controller |       09:43:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.552992 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.552996 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.553001 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.553005 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.553009 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.553013 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.553018 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.553022 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.553026 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.553030 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.553035 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.553039 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.553043 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.553047 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.553052 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.553058 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.553064 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.553068 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.553072 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.553077 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.553083 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.553087 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.553091 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.553095 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.553100 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.553104 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.553108 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.553112 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.553117 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.553121 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.553125 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.553130 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.553134 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.553138 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.553143 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.553147 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.553157 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.553162 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.553166 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.553170 | controller |       09:43:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.553175 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.553179 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.553183 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.553188 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.553192 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.553196 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.553200 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.553205 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.553209 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.553213 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.553217 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.553222 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.553226 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.553232 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.553237 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.553241 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.553245 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.553250 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.553254 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.553258 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.553263 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.553267 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.553271 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.553275 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.553280 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.553284 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.553288 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.553293 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.553297 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.553301 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.553305 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.553310 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.553314 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.553318 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.553323 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.553327 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.553331 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.553342 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.553346 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.553351 | controller |       09:43:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.553358 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.553362 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.553367 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.553371 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.553377 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.553381 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.553386 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.553390 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.553394 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.553398 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.553403 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.553407 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.553411 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.553415 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.553420 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.553424 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.553428 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.553432 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.553437 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.553441 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.553445 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.553450 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.553454 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.553458 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.553462 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.553467 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.553471 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.553475 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.553479 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.553484 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.553490 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.553494 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.553498 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.553503 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.553507 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.553511 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.553516 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.553522 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.553526 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.553544 | controller |       09:43:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.553548 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.553552 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.553557 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.553561 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.553565 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.553570 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.553574 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.553578 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.553583 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.553587 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.553591 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.553611 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.553615 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.553619 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.553626 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.553630 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.553634 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.553639 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.553643 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.553647 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.553651 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.553656 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.553660 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.553664 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.553668 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.553673 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.553677 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.553681 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.553686 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.553692 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.553696 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.553701 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.553705 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.553709 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.553713 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.553718 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.553722 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.553726 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.553730 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.553735 | controller |       09:43:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.553744 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.553749 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.553753 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.553758 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.553762 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.553766 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.553770 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.553775 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.553779 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.553783 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.553787 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.553792 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.553796 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.553800 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.553804 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.553809 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.553813 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.553817 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.553822 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.553826 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.553835 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.553839 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.553844 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.553848 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.553852 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.553856 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.553861 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.553865 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.553869 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.553873 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.553878 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.553882 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.553886 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.553891 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.553895 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.553899 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.553903 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.553908 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.553914 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.553918 | controller |       09:43:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.553922 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:43:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.553932 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.553936 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.553940 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.553947 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.553951 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.553955 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.553960 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.553964 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.553968 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.553973 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.553977 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.553981 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.553987 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.553991 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.553996 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.554000 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.554004 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.554009 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.554013 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.554017 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.554021 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.554026 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.554030 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.554034 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.554039 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.554043 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.554047 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.554051 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.554056 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.554060 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.554064 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.554068 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.554073 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.554077 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.554081 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.554086 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.554090 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:43:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.554094 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.554099 | controller |       09:43:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.554103 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.554107 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.554112 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.554121 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.554128 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.554132 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.554136 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.554141 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.554145 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.554149 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.554154 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.554158 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.554162 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.554169 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.554174 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.554178 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.554182 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.554187 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.554191 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.554195 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.554200 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.554204 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.554208 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.554212 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.554217 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.554221 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.554225 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.554230 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.554234 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.554238 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.554243 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.554247 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.554251 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.554255 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.554260 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.554264 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.554268 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.554275 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.554279 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.554283 | controller |       09:44:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.554288 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.554292 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.554296 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.554301 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.554310 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.554315 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.554319 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.554323 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.554328 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.554332 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.554336 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.554341 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.554345 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.554349 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.554353 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.554359 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.554364 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.554368 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.554372 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.554376 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.554381 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.554385 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.554389 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.554394 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.554398 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.554402 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.554406 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.554411 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.554415 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.554422 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.554427 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.554431 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.554435 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.554440 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.554444 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.554450 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.554454 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.554458 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.554464 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.554469 | controller |       09:44:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.554473 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.554477 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.554482 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.554486 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.554490 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.554500 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.554504 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.554508 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.554513 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.554517 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.554521 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.554525 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.554530 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.554534 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.554538 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.554543 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.554547 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.554551 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.554555 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.554560 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.554566 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.554570 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.554575 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.554579 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.554583 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.554587 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.554592 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.554608 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.554612 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.554617 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.554621 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.554625 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.554630 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.554634 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.554638 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.554643 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.554647 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.554651 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.554655 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.554660 | controller |       09:44:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.554664 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.554668 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.554673 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.554677 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.554681 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.554685 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.554690 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.554701 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.554706 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.554710 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.554715 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.554719 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.554725 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.554731 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.554735 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.554739 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.554744 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.554748 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.554752 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.554756 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.554762 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.554767 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.554771 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.554775 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.554779 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.554784 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.554788 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.554792 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.554796 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.554801 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.554806 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.554811 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.554815 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.554819 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.554824 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.554828 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.554832 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.554836 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.554841 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.554845 | controller |       09:44:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.554849 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.554854 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.554858 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.554862 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.554868 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.554873 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.554877 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.554881 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.554891 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.554895 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.554899 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.554917 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.554924 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.554928 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.554933 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.554938 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.554943 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.554949 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.554954 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.554959 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.554965 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.554970 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.554975 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.554981 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.554986 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.554991 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.554997 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.555002 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.555008 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.555014 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.555019 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.555025 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.555034 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.555040 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.555045 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.555051 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.555056 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.555064 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.555070 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.555076 | controller |       09:44:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.555081 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.555087 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.555093 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.555099 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.555106 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.555113 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.555120 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.555126 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.555131 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.555136 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.555152 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.555158 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.555164 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.555171 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.555176 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.555181 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.555187 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.555193 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.555198 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.555203 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.555209 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.555214 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.555220 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.555225 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.555230 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.555236 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.555241 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.555247 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.555252 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.555260 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.555266 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.555271 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.555277 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.555282 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.555288 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.555293 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.555299 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.555304 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.555310 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.555315 | controller |       09:44:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.555321 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.555326 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.555333 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.555340 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.555350 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.555357 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.555364 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.555370 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.555376 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.555381 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.555387 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.555404 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.555411 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.555417 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.555423 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.555429 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.555434 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.555440 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.555446 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.555453 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.555464 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.555470 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.555476 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.555481 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.555488 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.555494 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.555500 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.555505 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.555511 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.555517 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.555523 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.555529 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.555535 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.555541 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.555547 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.555553 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.555560 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.555565 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.555571 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.555577 | controller |       09:44:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.555584 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.555589 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.555609 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.555620 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.555626 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.555632 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.555638 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.555644 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.555650 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.555656 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.555662 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.555668 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.555684 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.555694 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.555700 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.555707 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.555713 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.555719 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.555725 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.555731 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.555737 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.555744 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.555750 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.555756 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.555762 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.555768 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.555774 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.555780 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.555790 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.555796 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.555802 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.555808 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.555814 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.555820 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.555827 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.555833 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.555839 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.555845 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.555851 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.555857 | controller |       09:44:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.555863 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.555869 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.555875 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.555882 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.555891 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.555897 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.555903 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.555910 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.555916 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.555922 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.555928 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.555934 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.555940 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.555946 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.555962 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.555972 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.555978 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.555984 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.555991 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.555997 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.556005 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.556011 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.556018 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.556024 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.556030 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.556036 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.556042 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.556048 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.556054 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.556060 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.556066 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.556073 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.556078 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.556084 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.556090 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.556096 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.556102 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.556112 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.556118 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.556124 | controller |       09:44:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.556130 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.556136 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.556142 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.556149 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.556155 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.556160 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.556166 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.556173 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.556179 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.556185 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.556194 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.556201 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.556207 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.556213 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.556219 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.556235 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.556242 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.556248 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.556254 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.556260 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.556266 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.556272 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.556278 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.556284 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.556290 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.556296 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.556302 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.556308 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.556314 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.556323 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.556330 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.556336 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.556342 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.556348 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.556354 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.556360 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.556366 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.556372 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.556379 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.556385 | controller |       09:44:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.556391 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.556397 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.556403 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.556409 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.556416 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.556422 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.556428 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.556434 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.556440 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.556446 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.556452 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.556458 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.556464 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.556470 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.556476 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.556482 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.556489 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.556508 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.556516 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.556522 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.556531 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.556538 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.556544 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.556550 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.556556 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.556563 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.556569 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.556579 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.556585 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.556591 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.556734 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.556742 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.556748 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.556754 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.556760 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.556767 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.556773 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.556778 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.556784 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.556790 | controller |       09:44:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.556795 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.556801 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.556806 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.556812 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.556818 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.556824 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.556830 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.556836 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.556841 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.556847 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.556853 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.556859 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.556864 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.556873 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.556879 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.556885 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.556891 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.556897 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.556915 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.556922 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.556928 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.556934 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.556940 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.556945 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.556951 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.556957 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.556963 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.556968 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.556974 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.556980 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.556985 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.556991 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.556997 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.557003 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.557009 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.557014 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.557020 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.557025 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.557031 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.557037 | controller |       09:44:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.557043 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.557048 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.557054 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.557060 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.557068 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.557074 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.557080 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.557086 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.557091 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.557100 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.557107 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.557112 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.557118 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.557124 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.557130 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.557136 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.557141 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.557147 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.557153 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.557168 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.557174 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.557180 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.557186 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.557192 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.557198 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.557204 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.557210 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.557216 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.557245 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.557251 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.557257 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.557262 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.557268 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.557274 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.557280 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.557285 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.557291 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.557302 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.557308 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.557314 | controller |       09:44:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.557320 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.557326 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.557332 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.557337 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.557346 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.557353 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.557359 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.557365 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.557370 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.557387 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.557394 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.557399 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.557405 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.557411 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.557416 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.557422 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.557428 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.557434 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.557440 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.557446 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.557465 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.557474 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.557480 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.557486 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.557492 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.557498 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.557504 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.557512 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.557518 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.557527 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.557533 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.557539 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.557544 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.557550 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.557556 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.557562 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.557568 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.557574 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.557580 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.557586 | controller |       09:44:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.557592 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.557615 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.557621 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.557627 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.557632 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.557638 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.557643 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.557649 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.557655 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.557661 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.557666 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.557671 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.557677 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.557682 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.557689 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.557694 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.557701 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.557706 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.557710 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.557714 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.557723 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.557727 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.557778 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.557785 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.557789 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.557794 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.557798 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.557802 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.557806 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.557811 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.557815 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.557819 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.557823 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.557828 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.557832 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.557836 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.557840 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.557845 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.557849 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.557853 | controller |       09:44:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.557858 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.557862 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.557866 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.557870 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.557875 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.557879 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.557883 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.557887 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.557895 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.557899 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.557903 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.557908 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.557912 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.557918 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.557922 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.557928 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.557933 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.557937 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.557941 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.557945 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.557950 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.557954 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.557958 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.557969 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.557974 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.557978 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.557982 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.557986 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.557991 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.557995 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.557999 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.558004 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.558008 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.558012 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.558016 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.558021 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.558025 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.558029 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.558034 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.558040 | controller |       09:44:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.558044 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.558048 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.558053 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.558057 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.558063 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.558067 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.558072 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.558076 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.558080 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.558084 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.558088 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.558093 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.558097 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.558101 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.558105 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.558110 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.558114 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.558118 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.558123 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.558127 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.558131 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.558135 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.558140 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.558144 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.558150 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.558162 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.558166 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.558171 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.558175 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.558179 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.558183 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.558188 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.558192 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.558196 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.558200 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.558205 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.558209 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.558215 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.558219 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.558224 | controller |       09:44:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.558228 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.558232 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.558237 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.558241 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.558245 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.558249 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.558254 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.558258 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.558262 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.558266 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.558271 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.558275 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.558279 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.558283 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.558288 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.558292 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.558296 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.558300 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.558305 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.558309 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.558313 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.558318 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.558322 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.558326 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.558330 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.558335 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.558345 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.558349 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.558354 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.558359 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.558367 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.558372 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.558376 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.558381 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.558385 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.558389 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.558393 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.558398 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.558402 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.558406 | controller |       09:44:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.558410 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.558415 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.558419 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.558425 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.558429 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.558433 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.558437 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.558442 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.558446 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.558450 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.558455 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.558459 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.558463 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.558467 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.558471 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.558476 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.558480 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.558484 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.558488 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.558493 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.558501 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.558505 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.558509 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.558514 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.558518 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.558522 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.558526 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.558531 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.558540 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.558545 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.558549 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.558554 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.558558 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.558562 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.558566 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.558571 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.558575 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.558579 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.558584 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.558588 | controller |       09:44:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.558592 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.558611 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.558616 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.558620 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.558624 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.558628 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.558641 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.558645 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.558650 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.558654 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.558658 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.558662 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.558667 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.558673 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.558678 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.558682 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.558686 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.558690 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.558695 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.558699 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.558703 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.558708 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.558712 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.558727 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.558732 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.558736 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.558740 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.558744 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.558749 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.558760 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.558765 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.558770 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.558774 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.558778 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.558782 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.558787 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.558791 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.558795 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.558799 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.558804 | controller |       09:44:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.558808 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.558812 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.558817 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.558821 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.558829 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.558833 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.558838 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.558842 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.558846 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.558850 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.558855 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.558859 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.558863 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.558868 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.558872 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.558876 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.558880 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.558885 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.558889 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.558893 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.558897 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.558902 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.558919 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.558925 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.558930 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.558934 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.558938 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.558943 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.558947 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.558951 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.558962 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.558967 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.558971 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.558975 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.558980 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.558984 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.558988 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.558995 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.558999 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.559003 | controller |       09:44:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.559008 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.559012 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.559018 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.559023 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.559027 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.559031 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.559036 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.559040 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.559044 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.559049 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.559053 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.559057 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.559061 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.559066 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.559070 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.559074 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.559078 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.559083 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.559087 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.559091 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.559095 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.559100 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.559104 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.559108 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.559112 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.559117 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.559121 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.559125 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.559129 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.559136 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.559140 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.559144 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.559154 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.559159 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.559163 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.559168 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.559172 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.559176 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.559180 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.559185 | controller |       09:44:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.559189 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.559193 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.559198 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.559202 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.559206 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.559210 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.559215 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.559219 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.559223 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.559227 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.559232 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.559236 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.559240 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.559245 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.559249 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.559255 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.559259 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.559264 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.559268 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.559272 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.559278 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.559284 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.559288 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.559292 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.559297 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.559301 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.559305 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.559310 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.559314 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.559318 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.559322 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.559327 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.559331 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.559341 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.559345 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.559349 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.559354 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.559358 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.559362 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.559367 | controller |       09:44:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.559371 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.559375 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.559379 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.559384 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.559388 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.559392 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.559396 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.559401 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.559405 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.559409 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.559413 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.559418 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.559422 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.559428 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.559432 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.559437 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.559441 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.559445 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.559449 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.559454 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.559461 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.559465 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.559469 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.559474 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.559478 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.559482 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.559486 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.559491 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.559495 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.559499 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.559503 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.559508 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.559512 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.559516 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.559526 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.559530 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.559534 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.559539 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.559543 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.559547 | controller |       09:44:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.559552 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.559558 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.559562 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.559566 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.559572 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.559577 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.559581 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.559585 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.559589 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.559606 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.559612 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.559617 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.559621 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.559625 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.559630 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.559634 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.559638 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.559643 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.559647 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.559651 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.559655 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.559660 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.559664 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.559668 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.559672 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.559677 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.559681 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.559685 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.559690 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.559694 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.559700 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.559704 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.559709 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.559713 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.559717 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.559722 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.559732 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.559738 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.559744 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.559749 | controller |       09:44:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.559753 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.559757 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.559762 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.559766 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.559770 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.559774 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.559779 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.559783 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.559787 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.559792 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.559796 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.559800 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.559804 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.559809 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.559813 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.559817 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.559823 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.559827 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.559832 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.559836 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.559840 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.559844 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.559849 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.559853 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.559857 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.559861 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.559866 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.559870 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.559874 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.559880 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.559885 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.559889 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.559893 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.559897 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.559902 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.559906 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.559910 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.559920 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.559924 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.559929 | controller |       09:44:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.559933 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.559937 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.559942 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.559946 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.559950 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.559954 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.559959 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.559963 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.559967 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.559971 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.559976 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.559980 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.559984 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.559988 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.559993 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.559997 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.560001 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.560005 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.560009 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.560014 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.560020 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.560024 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.560028 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.560033 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.560037 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.560041 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.560045 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.560050 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.560054 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.560058 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.560062 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.560067 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.560071 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.560075 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.560079 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.560084 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.560088 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.560092 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.560103 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.560108 | controller |       09:44:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.560112 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.560116 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.560121 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.560125 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.560131 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.560135 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.560140 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.560144 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.560148 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.560152 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.560157 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.560161 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.560165 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.560171 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.560175 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.560180 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.560184 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.560188 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.560192 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.560197 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.560201 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.560205 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.560210 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.560214 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.560218 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.560222 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.560227 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.560231 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.560235 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.560239 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.560244 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.560248 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.560252 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.560257 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.560261 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.560265 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.560270 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.560274 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.560278 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.560287 | controller |       09:44:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.560292 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.560297 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.560301 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.560305 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.560311 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.560316 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.560320 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.560324 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.560328 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.560333 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.560337 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.560341 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.560346 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.560350 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.560356 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.560360 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.560364 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.560368 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.560373 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.560377 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.560383 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.560387 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.560391 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.560396 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.560400 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.560404 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.560408 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.560413 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.560417 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.560421 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.560426 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.560430 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.560434 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.560438 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.560443 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.560447 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.560451 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.560458 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.560463 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.560467 | controller |       09:44:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.560471 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.560481 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.560485 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.560490 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.560494 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.560498 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.560503 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.560507 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.560511 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.560515 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.560519 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.560524 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.560528 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.560532 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.560536 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.560542 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.560547 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.560551 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.560555 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.560560 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.560564 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.560568 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.560572 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.560577 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.560581 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.560585 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.560589 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.560611 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.560625 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.560631 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.560635 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.560640 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.560644 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.560648 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.560653 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.560657 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.560661 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.560665 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.560672 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.560676 | controller |       09:44:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.560680 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.560685 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.560695 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.560700 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.560704 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.560709 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.560713 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.560717 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.560722 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.560726 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.560730 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.560734 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.560738 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.560743 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.560747 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.560751 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.560755 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.560760 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.560764 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.560768 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.560774 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.560778 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.560783 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.560787 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.560791 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.560795 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.560800 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.560804 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.560808 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.560812 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.560817 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.560821 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.560825 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.560829 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.560834 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.560838 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.560842 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.560847 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.560851 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.560855 | controller |       09:44:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.560859 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.560864 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.560868 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.560872 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.560881 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.560886 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.560890 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.560895 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.560899 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.560903 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.560907 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.560912 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.560916 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.560923 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.560928 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.560932 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.560936 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.560941 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.560945 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.560949 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.560954 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.560958 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.560962 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.560966 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.560971 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.560975 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.560979 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.560983 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.560988 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.560992 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.560998 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.561002 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.561007 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.561011 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.561015 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.561019 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.561024 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.561028 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.561032 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.561037 | controller |       09:44:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.561041 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.561045 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.561049 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.561054 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.561059 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.561069 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.561073 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.561078 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.561082 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.561086 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.561091 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.561095 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.561099 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.561103 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.561108 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.561112 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.561116 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.561120 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.561125 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.561129 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.561133 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.561138 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.561142 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.561146 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.561150 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.561155 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.561159 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.561163 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.561167 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.561171 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.561176 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.561180 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.561184 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.561189 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.561193 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.561199 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.561203 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.561209 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.561213 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.561218 | controller |       09:44:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.561222 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.561226 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.561230 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.561235 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.561239 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.561243 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.561247 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.561257 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.561261 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.561266 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.561270 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.561274 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.561278 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.561283 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.561287 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.561291 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.561295 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.561300 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.561304 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.561308 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.561314 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.561319 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.561323 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.561327 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.561331 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.561336 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.561340 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.561344 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.561348 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.561354 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.561359 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.561363 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.561367 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.561371 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.561376 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.561380 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.561384 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.561389 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.561393 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.561397 | controller |       09:44:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.561401 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.561406 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.561410 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.561424 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.561430 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.561435 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.561439 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.561443 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.561453 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.561457 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.561462 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.561466 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.561472 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.561476 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.561481 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.561485 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.561496 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.561500 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.561505 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.561509 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.561516 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.561521 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.561525 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.561529 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.561533 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.561538 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.561542 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.561546 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.561550 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.561555 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.561559 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.561563 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.561567 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.561572 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.561576 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.561580 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.561585 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.561589 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.561604 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.561610 | controller |       09:44:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.561615 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.561628 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.561633 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.561637 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.561641 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.561646 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.561650 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.561654 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.561659 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.561669 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.561674 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.561678 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.561683 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.561689 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.561693 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.561698 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.561702 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.561706 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.561711 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.561715 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.561719 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.561723 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.561728 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.561732 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.561736 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.561740 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.561745 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.561749 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.561753 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.561757 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.561762 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.561766 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.561772 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.561777 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.561781 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.561785 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.561790 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.561794 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.561798 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.561802 | controller |       09:44:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.561807 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.561811 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.561815 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.561819 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.561826 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.561832 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.561836 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.561840 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.561845 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.561849 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.561853 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.561863 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.561867 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.561872 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.561876 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.561880 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.561886 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.561891 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.561895 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.561899 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.561904 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.561908 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.561912 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.561917 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.561921 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.561925 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.561929 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.561934 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.561938 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.561942 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.561947 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.561951 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.561955 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.561959 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.561964 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.561968 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.561972 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.561986 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.561991 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.561995 | controller |       09:44:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.561999 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.562004 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.562008 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.562012 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.562016 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.562021 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.562025 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.562029 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.562034 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.562038 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.562042 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.562048 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.562058 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.562063 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.562067 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.562071 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.562076 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.562080 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.562084 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.562089 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.562100 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.562105 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.562109 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.562113 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.562118 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.562122 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.562126 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.562130 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.562135 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.562141 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.562145 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.562150 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.562154 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.562158 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.562163 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.562167 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.562171 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.562176 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.562180 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.562184 | controller |       09:44:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.562189 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.562193 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.562197 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.562201 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.562206 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.562210 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.562214 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.562219 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.562223 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.562227 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.562232 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.562236 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.562240 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.562244 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.562254 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.562258 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.562262 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.562267 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.562271 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.562275 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.562283 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.562287 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.562291 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.562296 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.562300 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.562304 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.562309 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.562313 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.562319 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.562323 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.562329 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.562333 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.562338 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.562342 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.562347 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.562351 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.562355 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.562360 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.562364 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.562368 | controller |       09:44:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.562373 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.562377 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.562381 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.562386 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.562390 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.562394 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.562399 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.562403 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.562407 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.562412 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.562416 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.562420 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.562425 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.562432 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.562437 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.562446 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.562451 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.562456 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.562460 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.562464 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.562469 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.562473 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.562477 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.562482 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.562486 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.562490 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.562495 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.562499 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.562504 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.562508 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.562512 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.562516 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.562521 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.562525 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.562530 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.562534 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.562538 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.562543 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.562547 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.562551 | controller |       09:44:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.562556 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.562560 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.562564 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.562569 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.562575 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.562579 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.562584 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.562588 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.562592 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.562610 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.562616 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.562621 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.562625 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.562630 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.562634 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.562638 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.562643 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.562653 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.562658 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.562662 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.562666 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.562671 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.562675 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.562680 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.562684 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.562688 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.562693 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.562697 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.562702 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.562708 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.562714 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.562720 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.562727 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.562731 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.562735 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.562740 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.562744 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.562751 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.562755 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.562760 | controller |       09:44:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.562764 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.562768 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.562773 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.562777 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.562784 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.562789 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.562793 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.562798 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.562802 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.562807 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.562811 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.562815 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.562820 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.562824 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.562828 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.562833 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.562837 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.562841 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.562852 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.562857 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.562861 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.562866 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.562870 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.562877 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.562881 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.562886 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.562890 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.562896 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.562900 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.562907 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.562926 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.562931 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.562935 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.562939 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.562944 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.562948 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.562952 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.562957 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.562961 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.562966 | controller |       09:44:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.562970 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.562974 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.562979 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.562983 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.562987 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.562992 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.562996 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.563000 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.563005 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.563009 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.563013 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.563018 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.563022 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.563026 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.563031 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.563035 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.563039 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.563044 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.563048 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.563058 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.563066 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.563070 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.563074 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.563079 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.563083 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.563088 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.563092 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.563096 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.563101 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.563105 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.563109 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.563114 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.563118 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.563122 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.563127 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.563131 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.563135 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.563140 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.563144 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.563148 | controller |       09:44:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.563153 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.563157 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.563161 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.563166 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.563170 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.563175 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.563179 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.563183 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.563188 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.563194 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.563198 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.563203 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.563207 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.563213 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.563217 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.563223 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.563228 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.563232 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.563236 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.563241 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.563252 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.563257 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.563261 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.563266 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.563270 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.563274 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.563279 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.563283 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.563287 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.563292 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.563296 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.563300 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.563305 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.563309 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.563313 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.563318 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.563322 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.563326 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.563331 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.563335 | controller |       09:44:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.563339 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.563344 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.563348 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.563352 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.563359 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.563363 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.563367 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.563372 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.563376 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.563380 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.563385 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.563389 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.563393 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.563398 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.563402 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.563406 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.563411 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.563415 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.563419 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.563424 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.563428 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.563432 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.563442 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.563446 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.563451 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.563457 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.563462 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.563467 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.563471 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.563476 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.563480 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.563484 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.563489 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.563493 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.563497 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.563502 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.563506 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.563514 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.563518 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.563523 | controller |       09:44:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.563527 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.563531 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.563536 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.563540 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.563544 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.563549 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.563553 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.563557 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.563562 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.563566 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.563570 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.563575 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.563579 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.563583 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.563588 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.563592 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.563611 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.563615 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.563620 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.563624 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.563628 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.563633 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.563637 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.563647 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.563652 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.563656 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.563660 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.563665 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.563669 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.563676 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.563682 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.563686 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.563691 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.563695 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.563699 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.563704 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.563708 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.563712 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.563717 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.563721 | controller |       09:44:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.563725 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.563730 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.563734 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.563738 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.563743 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.563747 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.563752 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.563756 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.563762 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.563766 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.563771 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.563777 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.563783 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.563789 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.563796 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.563802 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.563808 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.563814 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.563819 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.563823 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.563830 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.563834 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.563838 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.563843 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.563847 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.563858 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.563863 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.563867 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.563872 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.563876 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.563880 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.563885 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.563889 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.563893 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.563898 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.563902 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.563906 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.563911 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.563915 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.563919 | controller |       09:44:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.563924 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.563928 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.563932 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.563937 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.563941 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.563945 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.563950 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.563954 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.563958 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.563963 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.563967 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.563971 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.563976 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.563983 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.563987 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.563991 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.563996 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.564000 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.564004 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.564009 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.564013 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.564018 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.564022 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.564026 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.564033 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.564037 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.564047 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.564052 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.564056 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.564061 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.564065 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.564069 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.564074 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.564078 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.564083 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.564087 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.564091 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.564095 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.564102 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.564106 | controller |       09:44:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.564110 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.564115 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.564119 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.564123 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.564131 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.564135 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.564140 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.564144 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.564149 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.564153 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.564157 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.564161 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.564166 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.564170 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.564174 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.564179 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.564183 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.564187 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.564192 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.564196 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.564202 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.564206 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.564211 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.564215 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.564219 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.564224 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.564228 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.564238 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.564242 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.564247 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.564251 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.564255 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.564260 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.564264 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.564269 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.564273 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.564277 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.564283 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.564288 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.564292 | controller |       09:44:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.564297 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.564301 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.564305 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.564311 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.564316 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.564320 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.564324 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.564329 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.564334 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.564340 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.564344 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.564349 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.564355 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.564360 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.564364 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.564368 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.564373 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.564377 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.564381 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.564386 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.564390 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.564394 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.564399 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.564403 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.564407 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.564412 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.564416 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.564420 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.564425 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.564436 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.564441 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.564445 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.564450 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.564454 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.564459 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.564463 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.564468 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.564472 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.564476 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.564481 | controller |       09:44:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.564485 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:44:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.564489 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.564494 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.564498 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.564502 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.564507 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.564511 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.564515 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.564520 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.564524 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.564528 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.564533 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.564537 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.564541 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.564546 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.564552 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.564557 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.564562 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.564566 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.564571 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.564578 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.564582 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.564587 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.564593 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.564610 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.564615 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.564619 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.564623 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.564628 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.564632 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.564642 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.564647 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.564652 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.564656 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.564661 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.564665 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.564669 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.564674 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:44:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.564678 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.564682 | controller |       09:44:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.564687 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.564691 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.564695 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.564700 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.564704 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.564708 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.564713 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.564717 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.564722 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.564726 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.564730 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.564735 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.564739 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.564745 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.564750 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.564754 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.564758 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.564763 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.564767 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.564771 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.564776 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.564780 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.564784 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.564789 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.564793 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.564798 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.564802 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.564806 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.564810 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.564815 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.564819 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.564828 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.564833 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.564837 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.564842 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.564846 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.564851 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.564855 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.564860 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.564867 | controller |       09:45:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.564873 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.564879 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.564888 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.564895 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.564902 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.564906 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.564910 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.564915 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.564919 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.564923 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.564927 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.564932 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.564936 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.564940 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.564944 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.564949 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.564953 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.564957 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.564962 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.564966 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.564970 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.564974 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.564979 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.564983 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.564987 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.564991 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.564996 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.565000 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.565004 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.565009 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.565014 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.565019 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.565023 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.565034 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.565039 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.565043 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.565048 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.565054 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.565058 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.565063 | controller |       09:45:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.565067 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.565071 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.565076 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.565080 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.565084 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.565088 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.565093 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.565097 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.565101 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.565105 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.565110 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.565114 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.565118 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.565123 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.565127 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.565131 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.565135 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.565140 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.565144 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.565148 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.565154 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.565161 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.565165 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.565169 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.565174 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.565178 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.565182 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.565186 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.565191 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.565196 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.565201 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.565205 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.565209 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.565214 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.565224 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.565228 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.565233 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.565237 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.565241 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.565245 | controller |       09:45:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.565250 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.565254 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.565258 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.565263 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.565267 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.565271 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.565275 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.565280 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.565284 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.565288 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.565292 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.565297 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.565301 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.565305 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.565309 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.565315 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.565320 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.565324 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.565328 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.565333 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.565339 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.565343 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.565347 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.565351 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.565356 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.565360 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.565364 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.565368 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.565373 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.565377 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.565381 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.565385 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.565390 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.565394 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.565398 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.565408 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.565412 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.565417 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.565421 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.565425 | controller |       09:45:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.565430 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.565436 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.565440 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.565444 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.565450 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.565454 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.565458 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.565463 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.565467 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.565471 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.565476 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.565480 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.565484 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.565490 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.565494 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.565499 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.565503 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.565507 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.565512 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.565516 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.565520 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.565524 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.565529 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.565533 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.565537 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.565541 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.565546 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.565550 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.565554 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.565559 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.565563 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.565567 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.565571 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.565576 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.565580 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.565584 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.565589 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.565609 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.565616 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.565620 | controller |       09:45:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.565625 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.565629 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.565633 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.565637 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.565645 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.565650 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.565654 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.565658 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.565663 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.565667 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.565671 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.565675 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.565680 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.565684 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.565688 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.565692 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.565698 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.565703 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.565707 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.565711 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.565716 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.565720 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.565724 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.565728 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.565733 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.565737 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.565741 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.565745 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.565750 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.565754 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.565758 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.565762 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.565767 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.565771 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.565775 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.565780 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.565784 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.565790 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.565800 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.565805 | controller |       09:45:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.565809 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.565813 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.565818 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.565822 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.565826 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.565831 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.565835 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.565839 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.565843 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.565848 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.565852 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.565856 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.565860 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.565865 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.565869 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.565875 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.565880 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.565884 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.565888 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.565892 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.565897 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.565901 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.565905 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.565909 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.565914 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.565918 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.565922 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.565926 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.565931 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.565937 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.565942 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.565948 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.565954 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.565961 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.565967 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.565972 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.565977 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.565982 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.565988 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.566004 | controller |       09:45:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.566009 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.566013 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.566018 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.566022 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.566026 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.566031 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.566035 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.566039 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.566044 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.566058 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.566062 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.566066 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.566071 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.566075 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.566079 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.566083 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.566088 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.566092 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.566096 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.566101 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.566109 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.566114 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.566118 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.566122 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.566127 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.566131 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.566135 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.566140 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.566144 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.566148 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.566152 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.566157 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.566161 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.566165 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.566169 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.566174 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.566178 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.566182 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.566186 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.566191 | controller |       09:45:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.566195 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.566205 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.566210 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.566214 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.566218 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.566225 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.566229 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.566233 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.566238 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.566242 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.566246 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.566251 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.566255 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.566261 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.566265 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.566270 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.566275 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.566280 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.566284 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.566288 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.566293 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.566297 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.566301 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.566305 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.566310 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.566314 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.566318 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.566322 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.566327 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.566331 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.566335 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.566340 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.566346 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.566350 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.566354 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.566359 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.566363 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.566367 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.566371 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.566376 | controller |       09:45:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.566380 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.566384 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.566401 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.566405 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.566412 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.566416 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.566420 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.566425 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.566429 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.566433 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.566437 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.566442 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.566446 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.566450 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.566455 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.566459 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.566463 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.566467 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.566472 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.566476 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.566488 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.566492 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.566497 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.566501 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.566505 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.566509 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.566514 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.566518 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.566522 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.566526 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.566531 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.566535 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.566539 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.566544 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.566548 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.566552 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.566556 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.566562 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.566568 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.566573 | controller |       09:45:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.566577 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.566581 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.566585 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.566590 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.566613 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.566620 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.566624 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.566629 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.566633 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.566637 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.566642 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.566646 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.566650 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.566655 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.566659 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.566663 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.566668 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.566672 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.566676 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.566681 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.566685 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.566689 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.566693 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.566698 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.566702 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.566706 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.566711 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.566715 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.566719 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.566727 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.566731 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.566736 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.566740 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.566744 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.566749 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.566753 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.566757 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.566761 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.566766 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.566770 | controller |       09:45:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.566774 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.566779 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.566783 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.566787 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.566794 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.566803 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.566808 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.566812 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.566817 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.566821 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.566825 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.566829 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.566834 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.566840 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.566844 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.566849 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.566853 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.566857 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.566862 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.566866 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.566872 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.566876 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.566881 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.566885 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.566889 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.566894 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.566898 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.566902 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.566906 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.566923 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.566929 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.566934 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.566938 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.566942 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.566946 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.566951 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.566957 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.566963 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.566968 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.566975 | controller |       09:45:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.566980 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.566986 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.566992 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.566997 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.567002 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.567006 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.567017 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.567021 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.567026 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.567030 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.567035 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.567039 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.567043 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.567050 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.567054 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.567059 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.567063 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.567067 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.567071 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.567076 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.567082 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.567087 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.567091 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.567095 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.567099 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.567104 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.567108 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.567112 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.567116 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.567121 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.567125 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.567129 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.567133 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.567138 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.567142 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.567148 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.567152 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.567157 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.567161 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.567165 | controller |       09:45:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.567169 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.567174 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.567178 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.567182 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.567188 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.567193 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.567197 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.567201 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.567211 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.567215 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.567219 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.567224 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.567228 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.567232 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.567237 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.567243 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.567247 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.567252 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.567256 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.567260 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.567264 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.567269 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.567273 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.567277 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.567282 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.567286 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.567290 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.567294 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.567299 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.567303 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.567307 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.567311 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.567316 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.567320 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.567324 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.567329 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.567333 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.567339 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.567344 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.567348 | controller |       09:45:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.567352 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.567357 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.567361 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.567365 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.567369 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.567374 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.567378 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.567382 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.567387 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.567396 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.567400 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.567405 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.567414 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.567419 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.567423 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.567427 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.567432 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.567436 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.567440 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.567444 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.567449 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.567453 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.567457 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.567461 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.567466 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.567470 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.567474 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.567478 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.567483 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.567492 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.567496 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.567500 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.567505 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.567509 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.567513 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.567517 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.567522 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.567526 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.567530 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.567535 | controller |       09:45:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.567539 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.567543 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.567547 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.567552 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.567556 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.567560 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.567565 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.567569 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.567573 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.567577 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.567582 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.567592 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.567610 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.567615 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.567619 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.567623 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.567628 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.567632 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.567636 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.567640 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.567647 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.567651 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.567655 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.567659 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.567664 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.567668 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.567672 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.567676 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.567681 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.567685 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.567691 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.567696 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.567701 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.567706 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.567710 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.567714 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.567718 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.567723 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.567727 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.567731 | controller |       09:45:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.567736 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.567740 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.567744 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.567748 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.567753 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.567757 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.567761 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.567768 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.567773 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.567777 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.567782 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.567786 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.567796 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.567803 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.567807 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.567812 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.567816 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.567820 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.567824 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.567829 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.567833 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.567837 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.567841 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.567846 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.567850 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.567854 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.567858 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.567863 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.567867 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.567871 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.567875 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.567880 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.567884 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.567888 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.567893 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.567897 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.567901 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.567905 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.567910 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.567914 | controller |       09:45:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.567918 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.567922 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.567927 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.567931 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.567937 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.567954 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.567959 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.567963 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.567967 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.567971 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.567976 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.567982 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.567986 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.567990 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.568000 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.568005 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.568009 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.568013 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.568018 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.568022 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.568028 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.568032 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.568037 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.568041 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.568045 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.568050 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.568054 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.568058 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.568062 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.568067 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.568071 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.568075 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.568080 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.568084 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.568088 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.568092 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.568097 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.568103 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.568107 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.568111 | controller |       09:45:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.568116 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.568120 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.568124 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.568129 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.568134 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.568139 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.568143 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.568147 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.568152 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.568156 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.568160 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.568164 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.568169 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.568173 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.568177 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.568187 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.568191 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.568195 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.568200 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.568204 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.568208 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.568213 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.568217 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.568221 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.568226 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.568230 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.568234 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.568238 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.568244 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.568250 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.568254 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.568259 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.568263 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.568267 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.568272 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.568276 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.568280 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.568285 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.568289 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.568293 | controller |       09:45:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.568297 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.568302 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.568306 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.568310 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.568314 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.568319 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.568323 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.568327 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.568333 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.568338 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.568343 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.568347 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.568353 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.568358 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.568362 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.568367 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.568371 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.568380 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.568384 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.568389 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.568395 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.568399 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.568403 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.568408 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.568412 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.568416 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.568420 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.568425 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.568429 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.568433 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.568437 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.568442 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.568446 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.568450 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.568455 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.568459 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.568463 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.568467 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.568473 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.568477 | controller |       09:45:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.568482 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.568486 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.568490 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.568495 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.568499 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.568503 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.568507 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.568512 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.568516 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.568520 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.568526 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.568530 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.568535 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.568541 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.568545 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.568551 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.568555 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.568560 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.568569 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.568574 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.568578 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.568582 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.568587 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.568591 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.568608 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.568612 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.568617 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.568621 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.568625 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.568629 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.568634 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.568638 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.568642 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.568646 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.568651 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.568655 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.568659 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.568663 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.568668 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.568672 | controller |       09:45:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.568676 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.568680 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.568685 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.568689 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.568695 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.568699 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.568704 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.568708 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.568712 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.568717 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.568721 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.568725 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.568729 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.568734 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.568738 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.568742 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.568746 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.568751 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.568755 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.568765 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.568769 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.568774 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.568778 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.568782 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.568787 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.568791 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.568795 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.568803 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.568807 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.568811 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.568816 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.568820 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.568824 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.568828 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.568833 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.568837 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.568841 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.568847 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.568852 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.568856 | controller |       09:45:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.568860 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.568865 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.568869 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.568873 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.568877 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.568882 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.568886 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.568890 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.568895 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.568899 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.568903 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.568907 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.568912 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.568916 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.568920 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.568925 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.568929 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.568933 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.568937 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.568942 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.568953 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.568958 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.568962 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.568966 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.568971 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.568975 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.568979 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.568984 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.568988 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.568994 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.569000 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.569006 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.569012 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.569018 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.569024 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.569030 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.569034 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.569039 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.569043 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.569047 | controller |       09:45:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.569052 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.569056 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.569060 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.569097 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.569104 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.569108 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.569113 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.569117 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.569121 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.569129 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.569134 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.569138 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.569142 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.569147 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.569153 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.569159 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.569165 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.569171 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.569177 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.569183 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.569191 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.569204 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.569208 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.569213 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.569217 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.569222 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.569226 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.569230 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.569234 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.569239 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.569243 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.569247 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.569252 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.569256 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.569260 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.569265 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.569269 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.569273 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.569277 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.569282 | controller |       09:45:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.569286 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.569291 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.569295 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.569299 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.569304 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.569308 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.569312 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.569316 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.569321 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.569325 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.569329 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.569334 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.569338 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.569344 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.569349 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.569353 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.569357 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.569362 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.569366 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.569370 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.569374 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.569379 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.569383 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.569393 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.569397 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.569404 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.569408 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.569413 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.569417 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.569421 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.569425 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.569430 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.569434 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.569438 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.569443 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.569447 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.569451 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.569456 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.569460 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.569464 | controller |       09:45:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.569468 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.569473 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.569477 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.569481 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.569489 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.569493 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.569498 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.569502 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.569506 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.569511 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.569515 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.569519 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.569523 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.569528 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.569532 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.569536 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.569540 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.569545 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.569549 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.569553 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.569558 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.569562 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.569566 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.569570 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.569580 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.569585 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.569589 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.569605 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.569612 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.569616 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.569629 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.569633 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.569637 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.569642 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.569646 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.569650 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.569655 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.569662 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.569666 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.569671 | controller |       09:45:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.569675 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.569679 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.569684 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.569688 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.569692 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.569697 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.569701 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.569705 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.569712 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.569716 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.569721 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.569725 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.569729 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.569733 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.569738 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.569742 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.569746 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.569750 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.569755 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.569759 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.569763 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.569768 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.569772 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.569776 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.569781 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.569785 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.569796 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.569800 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.569805 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.569811 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.569815 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.569820 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.569824 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.569828 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.569833 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.569837 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.569841 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.569846 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.569850 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.569854 | controller |       09:45:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.569858 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.569863 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.569867 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.569871 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.569875 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.569880 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.569884 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.569888 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.569893 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.569897 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.569901 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.569906 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.569910 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.569914 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.569918 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.569925 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.569929 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.569933 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.569938 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.569942 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.569951 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.569955 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.569959 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.569964 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.569969 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.569974 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.569978 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.569988 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.569992 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.569997 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.570001 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.570006 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.570010 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.570014 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.570019 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.570023 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.570027 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.570031 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.570036 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.570040 | controller |       09:45:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.570044 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.570049 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.570053 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.570057 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.570062 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.570066 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.570070 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.570075 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.570079 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.570083 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.570087 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.570092 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.570096 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.570102 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.570107 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.570111 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.570115 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.570119 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.570124 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.570128 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.570132 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.570137 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.570141 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.570145 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.570149 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.570154 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.570158 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.570162 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.570167 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.570176 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.570180 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.570185 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.570189 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.570194 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.570198 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.570202 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.570206 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.570211 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.570215 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.570219 | controller |       09:45:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.570224 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.570228 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.570232 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.570238 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.570244 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.570249 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.570253 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.570258 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.570262 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.570266 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.570270 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.570275 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.570279 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.570283 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.570288 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.570292 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.570296 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.570301 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.570305 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.570309 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.570314 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.570318 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.570322 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.570326 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.570331 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.570335 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.570339 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.570344 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.570348 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.570352 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.570363 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.570368 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.570372 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.570376 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.570381 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.570385 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.570389 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.570395 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.570400 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.570404 | controller |       09:45:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.570408 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.570413 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.570417 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.570421 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.570425 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.570430 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.570434 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.570438 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.570443 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.570447 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.570451 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.570455 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.570460 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.570464 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.570468 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.570473 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.570477 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.570481 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.570486 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.570490 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.570494 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.570498 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.570503 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.570509 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.570513 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.570517 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.570522 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.570526 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.570530 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.570536 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.570541 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.570550 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.570555 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.570559 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.570563 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.570568 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.570572 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.570576 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.570581 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.570585 | controller |       09:45:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.570589 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.570604 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.570611 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.570615 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.570619 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.570625 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.570630 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.570634 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.570638 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.570643 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.570647 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.570651 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.570656 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.570660 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.570664 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.570668 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.570673 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.570677 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.570681 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.570686 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.570692 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.570698 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.570702 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.570707 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.570711 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.570715 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.570719 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.570724 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.570728 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.570732 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.570736 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.570741 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.570745 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.570756 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.570760 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.570764 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.570769 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.570773 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.570777 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.570782 | controller |       09:45:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.570786 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.570790 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.570796 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.570801 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.570806 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.570810 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.570815 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.570819 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.570823 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.570827 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.570832 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.570836 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.570840 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.570848 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.570852 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.570857 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.570861 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.570865 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.570870 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.570874 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.570880 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.570884 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.570889 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.570893 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.570897 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.570902 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.570906 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.570911 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.570930 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.570935 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.570939 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.570943 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.570947 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.570952 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.570962 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.570966 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.570971 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.570975 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.570979 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.570984 | controller |       09:45:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.570988 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.570992 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.570997 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.571001 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.571007 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.571012 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.571016 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.571020 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.571024 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.571029 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.571033 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.571037 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.571041 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.571046 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.571050 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.571054 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.571059 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.571063 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.571067 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.571071 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.571076 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.571082 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.571086 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.571091 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.571095 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.571099 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.571104 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.571108 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.571112 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.571116 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.571121 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.571125 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.571129 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.571134 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.571138 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.571147 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.571152 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.571158 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.571162 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.571167 | controller |       09:45:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.571171 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.571175 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.571180 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.571184 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.571188 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.571192 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.571197 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.571201 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.571205 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.571210 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.571214 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.571218 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.571223 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.571227 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.571231 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.571237 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.571242 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.571246 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.571250 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.571254 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.571259 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.571263 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.571267 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.571272 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.571276 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.571280 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.571284 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.571289 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.571293 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.571299 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.571303 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.571307 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.571312 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.571316 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.571322 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.571328 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.571334 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.571348 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.571354 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.571359 | controller |       09:45:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.571363 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.571370 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.571374 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.571379 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.571383 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.571387 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.571392 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.571396 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.571400 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.571404 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.571409 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.571413 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.571417 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.571421 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.571426 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.571430 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.571434 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.571439 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.571443 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.571447 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.571453 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.571458 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.571462 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.571466 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.571470 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.571475 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.571479 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.571483 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.571487 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.571492 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.571496 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.571500 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.571505 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.571509 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.571513 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.571517 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.571522 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.571526 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.571536 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.571540 | controller |       09:45:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.571545 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.571549 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.571553 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.571557 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.571562 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.571566 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.571570 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.571574 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.571579 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.571583 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.571587 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.571591 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.571609 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.571616 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.571629 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.571633 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.571640 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.571644 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.571649 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.571653 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.571657 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.571661 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.571666 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.571670 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.571674 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.571678 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.571683 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.571687 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.571691 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.571695 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.571701 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.571706 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.571710 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.571714 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.571718 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.571723 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.571727 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.571731 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.571736 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.571747 | controller |       09:45:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.571751 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.571756 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.571760 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.571764 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.571771 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.571775 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.571779 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.571784 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.571788 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.571792 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.571796 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.571801 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.571805 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.571809 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.571813 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.571818 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.571822 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.571826 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.571831 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.571835 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.571841 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.571846 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.571850 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.571854 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.571858 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.571863 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.571867 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.571871 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.571876 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.571880 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.571884 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.571889 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.571893 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.571897 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.571901 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.571906 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.571910 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.571916 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.571920 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.571925 | controller |       09:45:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.571932 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.571942 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.571946 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.571951 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.571955 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.571959 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.571964 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.571968 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.571972 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.571976 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.571981 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.571985 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.571989 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.571994 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.571998 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.572002 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.572006 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.572011 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.572015 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.572019 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.572023 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.572028 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.572032 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.572036 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.572041 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.572045 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.572049 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.572053 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.572058 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.572064 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.572068 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.572073 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.572077 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.572081 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.572085 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.572090 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.572094 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.572098 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.572102 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.572107 | controller |       09:45:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.572111 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.572115 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.572125 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.572129 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.572136 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.572140 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.572144 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.572148 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.572153 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.572157 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.572161 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.572165 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.572170 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.572174 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.572180 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.572184 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.572188 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.572193 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.572197 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.572201 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.572207 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.572211 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.572216 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.572220 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.572224 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.572228 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.572233 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.572237 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.572241 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.572245 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.572250 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.572254 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.572258 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.572263 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.572267 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.572271 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.572275 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.572280 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.572284 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.572288 | controller |       09:45:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.572292 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.572297 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.572301 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.572305 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.572315 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.572319 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.572324 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.572328 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.572332 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.572337 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.572341 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.572345 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.572349 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.572355 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.572360 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.572364 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.572368 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.572373 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.572377 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.572381 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.572385 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.572390 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.572394 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.572398 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.572404 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.572410 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.572416 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.572422 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.572428 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.572434 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.572440 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.572445 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.572449 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.572454 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.572458 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.572462 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.572467 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.572471 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.572478 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.572482 | controller |       09:45:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.572486 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.572491 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.572495 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.572499 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.572506 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.572516 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.572521 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.572525 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.572530 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.572534 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.572538 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.572542 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.572547 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.572551 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.572555 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.572561 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.572566 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.572570 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.572574 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.572579 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.572583 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.572587 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.572592 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.572609 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.572613 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.572618 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.572622 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.572626 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.572631 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.572635 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.572639 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.572645 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.572650 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.572654 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.572658 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.572662 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.572667 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.572673 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.572677 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.572682 | controller |       09:45:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.572686 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.572690 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.572694 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.572699 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.572703 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.572707 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.572718 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.572722 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.572727 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.572731 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.572735 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.572740 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.572744 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.572750 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.572754 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.572759 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.572763 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.572767 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.572771 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.572776 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.572781 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.572786 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.572790 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.572794 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.572799 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.572803 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.572807 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.572811 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.572816 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.572822 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.572826 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.572830 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.572835 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.572839 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.572843 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.572847 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.572852 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.572856 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.572862 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.572866 | controller |       09:45:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.572871 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.572875 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.572879 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.572883 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.572888 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.572892 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.572896 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.572900 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.572910 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.572914 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.572919 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.572923 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.572927 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.572932 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.572936 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.572940 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.572945 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.572949 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.572953 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.572957 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.572966 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.572970 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.572975 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.572979 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.572983 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.572987 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.572992 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.572996 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.573000 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.573005 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.573010 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.573015 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.573019 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.573023 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.573028 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.573033 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.573038 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.573042 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.573046 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.573051 | controller |       09:45:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.573055 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.573059 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.573063 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.573068 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.573072 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.573076 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.573081 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.573085 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.573089 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.573099 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.573103 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.573108 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.573112 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.573118 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.573123 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.573127 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.573131 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.573136 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.573140 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.573144 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.573148 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.573153 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.573157 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.573161 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.573166 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.573170 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.573174 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.573178 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.573183 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.573187 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.573191 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.573195 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.573200 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.573204 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.573208 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.573212 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.573217 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.573221 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.573225 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.573230 | controller |       09:45:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.573234 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.573238 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.573243 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.573247 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.573253 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.573257 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.573261 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.573265 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.573270 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.573274 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.573278 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.573289 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.573294 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.573298 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.573303 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.573307 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.573311 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.573315 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.573320 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.573324 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.573328 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.573333 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.573337 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.573341 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.573345 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.573350 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.573354 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.573358 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.573363 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.573367 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.573371 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.573375 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.573380 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.573384 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.573388 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.573393 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.573397 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.573403 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.573407 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.573411 | controller |       09:45:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.573416 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.573420 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.573424 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.573429 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.573434 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.573439 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.573443 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.573447 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.573452 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.573456 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.573460 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.573465 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.573474 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.573478 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.573482 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.573488 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.573494 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.573500 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.573506 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.573512 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.573518 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.573524 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.573528 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.573532 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.573537 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.573541 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.573545 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.573549 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.573554 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.573560 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.573565 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.573569 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.573575 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.573579 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.573584 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.573588 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.573592 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.573629 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.573634 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.573638 | controller |       09:45:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.573643 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.573647 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.573651 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.573655 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.573660 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.573664 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.573668 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.573672 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.573677 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.573681 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.573685 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.573690 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.573694 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.573706 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.573711 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.573715 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.573720 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.573724 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.573728 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.573733 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.573741 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.573745 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.573750 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.573754 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.573758 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.573762 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.573767 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.573771 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.573775 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.573779 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.573784 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.573788 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.573792 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.573797 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.573801 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.573805 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.573809 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.573813 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.573818 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.573822 | controller |       09:45:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.573826 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.573830 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.573835 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.573839 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.573843 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.573848 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.573852 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.573856 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.573860 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.573865 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.573869 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.573875 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.573879 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.573885 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.573889 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.573902 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.573907 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.573911 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.573915 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.573919 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.573924 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.573928 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.573932 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.573937 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.573941 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.573945 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.573949 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.573954 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.573958 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.573962 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.573966 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.573971 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.573975 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.573979 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.573983 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.573988 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.573992 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.573996 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.574001 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.574005 | controller |       09:45:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.574009 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.574013 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.574018 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.574022 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.574030 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.574034 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.574038 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.574043 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.574047 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.574051 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.574055 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.574060 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.574064 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.574068 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.574072 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.574077 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.574087 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.574092 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.574096 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.574100 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.574105 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.574109 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.574113 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.574118 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.574122 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.574126 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.574130 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.574135 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.574142 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.574146 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.574151 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.574155 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.574159 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.574164 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.574168 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.574172 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.574176 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.574182 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.574187 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.574191 | controller |       09:45:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.574195 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:45:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.574200 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.574204 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.574208 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.574212 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.574217 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.574221 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.574225 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.574229 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.574234 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.574238 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.574242 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.574246 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.574250 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.574255 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.574259 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.574263 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.574267 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.574278 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.574283 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.574287 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.574291 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.574296 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.574300 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.574304 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.574309 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.574313 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.574317 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.574321 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.574327 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.574333 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.574338 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.574342 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.574346 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.574351 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.574355 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.574359 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.574363 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:45:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.574368 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.574372 | controller |       09:45:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.574376 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.574380 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.574385 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.574389 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.574393 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.574397 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.574402 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.574406 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.574410 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.574414 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.574420 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.574425 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.574429 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.574433 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.574437 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.574442 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.574446 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.574450 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.574454 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.574466 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.574472 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.574476 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.574481 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.574485 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.574489 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.574494 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.574498 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.574502 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.574506 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.574511 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.574515 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.574519 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.574523 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.574528 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.574532 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.574536 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.574540 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.574545 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.574549 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.574553 | controller |       09:46:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.574557 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.574562 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.574567 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.574573 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.574579 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.574585 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.574591 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.574610 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.574615 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.574619 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.574624 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.574628 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.574632 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.574639 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.574643 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.574647 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.574652 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.574656 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.574660 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.574665 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.574679 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.574684 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.574688 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.574692 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.574697 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.574701 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.574705 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.574711 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.574715 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.574719 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.574724 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.574728 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.574732 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.574737 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.574741 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.574745 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.574749 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.574754 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.574758 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.574762 | controller |       09:46:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.574766 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.574771 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.574775 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.574779 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.574787 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.574791 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.574795 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.574800 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.574804 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.574808 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.574812 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.574817 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.574821 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.574825 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.574829 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.574834 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.574838 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.574842 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.574846 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.574851 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.574855 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.574865 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.574870 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.574874 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.574878 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.574883 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.574887 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.574891 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.574895 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.574900 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.574904 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.574908 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.574912 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.574930 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.574935 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.574940 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.574944 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.574950 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.574955 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.574959 | controller |       09:46:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.574963 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.574967 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.574972 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.574976 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.574980 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.574986 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.574991 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.574995 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.574999 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.575004 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.575008 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.575012 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.575016 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.575020 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.575025 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.575029 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.575035 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.575039 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.575043 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.575048 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.575052 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.575056 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.575060 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.575074 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.575079 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.575083 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.575088 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.575092 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.575096 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.575102 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.575107 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.575111 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.575115 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.575120 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.575124 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.575128 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.575132 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.575137 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.575141 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.575145 | controller |       09:46:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.575150 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.575154 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.575158 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.575162 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.575167 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.575171 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.575175 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.575179 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.575184 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.575188 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.575192 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.575196 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.575201 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.575205 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.575209 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.575215 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.575220 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.575225 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.575229 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.575233 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.575239 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.575243 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.575248 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.575252 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.575262 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.575268 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.575273 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.575277 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.575281 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.575286 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.575290 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.575294 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.575298 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.575303 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.575307 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.575311 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.575315 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.575320 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.575324 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.575328 | controller |       09:46:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.575333 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.575337 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.575341 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.575345 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.575349 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.575354 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.575358 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.575362 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.575366 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.575371 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.575375 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.575379 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.575383 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.575389 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.575394 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.575398 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.575402 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.575406 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.575411 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.575415 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.575419 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.575423 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.575428 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.575432 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.575436 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.575440 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.575450 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.575454 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.575459 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.575463 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.575467 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.575471 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.575476 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.575480 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.575484 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.575489 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.575493 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.575497 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.575501 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.575506 | controller |       09:46:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.575510 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.575514 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.575518 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.575523 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.575529 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.575533 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.575537 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.575542 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.575547 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.575552 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.575556 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.575560 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.575565 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.575569 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.575573 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.575577 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.575581 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.575586 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.575590 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.575606 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.575614 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.575619 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.575623 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.575627 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.575631 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.575636 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.575640 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.575651 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.575658 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.575664 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.575672 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.575679 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.575684 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.575689 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.575694 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.575700 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.575706 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.575715 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.575719 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.575723 | controller |       09:46:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.575728 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.575732 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.575736 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.575740 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.575745 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.575749 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.575753 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.575757 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.575762 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.575766 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.575770 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.575774 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.575778 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.575783 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.575787 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.575791 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.575795 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.575800 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.575804 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.575808 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.575812 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.575817 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.575821 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.575825 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.575832 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.575837 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.575841 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.575845 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.575849 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.575862 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.575867 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.575871 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.575875 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.575880 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.575884 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.575888 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.575892 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.575897 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.575901 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.575905 | controller |       09:46:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.575909 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.575914 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.575918 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.575922 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.575926 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.575931 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.575935 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.575939 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.575943 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.575948 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.575952 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.575956 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.575960 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.575965 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.575969 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.575973 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.575977 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.575982 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.575986 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.575990 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.575996 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.576000 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.576004 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.576009 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.576013 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.576017 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.576022 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.576026 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.576030 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.576034 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.576044 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.576048 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.576053 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.576057 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.576061 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.576066 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.576070 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.576074 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.576079 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.576083 | controller |       09:46:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.576087 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.576091 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.576096 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.576102 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.576113 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.576117 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.576122 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.576126 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.576130 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.576134 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.576139 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.576143 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.576147 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.576153 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.576157 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.576162 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.576166 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.576170 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.576174 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.576179 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.576183 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.576187 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.576192 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.576196 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.576200 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.576204 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.576209 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.576213 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.576217 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.576221 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.576226 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.576236 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.576240 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.576245 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.576249 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.576253 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.576257 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.576262 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.576266 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.576270 | controller |       09:46:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.576275 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.576279 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.576283 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.576287 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.576294 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.576298 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.576302 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.576307 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.576311 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.576315 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.576319 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.576324 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.576328 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.576332 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.576336 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.576341 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.576345 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.576349 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.576353 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.576358 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.576362 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.576366 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.576371 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.576376 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.576381 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.576385 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.576389 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.576394 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.576398 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.576402 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.576407 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.576411 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.576415 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.576425 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.576429 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.576433 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.576438 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.576444 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.576448 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.576452 | controller |       09:46:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.576457 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.576461 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.576465 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.576469 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.576474 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.576478 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.576482 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.576487 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.576491 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.576495 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.576499 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.576504 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.576508 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.576512 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.576516 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.576522 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.576527 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.576531 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.576535 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.576539 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.576545 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.576549 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.576554 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.576558 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.576562 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.576567 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.576571 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.576575 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.576579 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.576585 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.576590 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.576606 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.576613 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.576617 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.576627 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.576632 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.576636 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.576640 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.576645 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.576649 | controller |       09:46:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.576653 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.576658 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.576664 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.576669 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.576673 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.576677 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.576682 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.576686 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.576690 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.576695 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.576699 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.576703 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.576708 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.576712 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.576716 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.576721 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.576725 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.576729 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.576734 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.576738 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.576744 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.576748 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.576753 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.576758 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.576765 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.576771 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.576776 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.576781 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.576785 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.576790 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.576794 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.576798 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.576803 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.576807 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.576811 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.576822 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.576826 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.576831 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.576835 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.576840 | controller |       09:46:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.576844 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.576848 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.576853 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.576857 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.576861 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.576866 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.576870 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.576875 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.576879 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.576883 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.576888 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.576892 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.576896 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.576903 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.576907 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.576911 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.576916 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.576920 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.576925 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.576929 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.576933 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.576940 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.576944 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.576949 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.576953 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.576958 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.576962 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.576966 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.576970 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.576975 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.576981 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.576985 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.576989 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.576994 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.576998 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.577002 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.577007 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.577017 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.577021 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.577026 | controller |       09:46:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.577030 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.577035 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.577039 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.577043 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.577049 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.577054 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.577058 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.577062 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.577067 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.577071 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.577076 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.577080 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.577084 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.577088 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.577093 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.577097 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.577101 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.577106 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.577110 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.577114 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.577119 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.577123 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.577127 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.577132 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.577136 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.577140 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.577145 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.577149 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.577154 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.577158 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.577162 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.577166 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.577171 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.577175 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.577180 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.577184 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.577188 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.577195 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.577208 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.577213 | controller |       09:46:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.577217 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.577223 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.577228 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.577232 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.577236 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.577241 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.577245 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.577249 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.577254 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.577258 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.577262 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.577267 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.577271 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.577275 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.577280 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.577284 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.577288 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.577293 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.577297 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.577301 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.577306 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.577310 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.577314 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.577319 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.577323 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.577327 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.577332 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.577336 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.577340 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.577347 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.577351 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.577356 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.577360 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.577364 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.577369 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.577373 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.577377 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.577382 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.577386 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.577396 | controller |       09:46:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.577400 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.577405 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.577409 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.577414 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.577420 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.577424 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.577428 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.577433 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.577437 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.577441 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.577446 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.577450 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.577454 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.577459 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.577463 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.577469 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.577475 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.577479 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.577484 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.577488 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.577496 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.577500 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.577505 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.577509 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.577513 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.577518 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.577522 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.577526 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.577531 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.577535 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.577539 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.577544 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.577548 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.577552 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.577557 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.577561 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.577565 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.577570 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.577574 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.577578 | controller |       09:46:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.577589 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.577606 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.577612 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.577617 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.577621 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.577626 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.577630 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.577634 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.577639 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.577643 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.577647 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.577652 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.577656 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.577663 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.577667 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.577671 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.577676 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.577680 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.577684 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.577694 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.577766 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.577772 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.577776 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.577780 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.577785 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.577789 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.577793 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.577798 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.577802 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.577806 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.577811 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.577815 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.577820 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.577828 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.577836 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.577842 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.577849 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.577856 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.577860 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.577865 | controller |       09:46:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.577874 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.577878 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.577887 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.577896 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.577919 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.577930 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.577934 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.577938 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.577943 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.577947 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.577951 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.577956 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.577960 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.577964 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.577969 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.577981 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.577987 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.577999 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.578007 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.578020 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.578025 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.578029 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.578037 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.578049 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.578054 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.578058 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.578063 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.578067 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.578071 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.578076 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.578080 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.578084 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.578089 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.578093 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.578097 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.578102 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.578106 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.578112 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.578117 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.578121 | controller |       09:46:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.578125 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.578130 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.578134 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.578144 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.578149 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.578153 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.578157 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.578162 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.578166 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.578170 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.578175 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.578179 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.578183 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.578188 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.578194 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.578198 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.578203 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.578207 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.578212 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.578216 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.578220 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.578225 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.578229 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.578233 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.578237 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.578242 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.578246 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.578250 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.578255 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.578261 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.578265 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.578269 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.578274 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.578278 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.578282 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.578287 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.578291 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.578295 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.578299 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.578304 | controller |       09:46:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.578308 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.578312 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.578317 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.578321 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.578325 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.578335 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.578340 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.578344 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.578349 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.578353 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.578357 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.578362 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.578366 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.578370 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.578375 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.578379 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.578383 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.578387 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.578392 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.578396 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.578404 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.578408 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.578413 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.578417 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.578421 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.578426 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.578430 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.578434 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.578438 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.578443 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.578449 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.578453 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.578458 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.578462 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.578466 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.578471 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.578475 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.578479 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.578485 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.578489 | controller |       09:46:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.578494 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.578498 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.578502 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.578507 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.578511 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.578516 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.578525 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.578530 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.578534 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.578539 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.578543 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.578548 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.578552 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.578558 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.578562 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.578567 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.578571 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.578576 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.578580 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.578584 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.578590 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.578614 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.578621 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.578626 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.578630 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.578635 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.578639 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.578643 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.578648 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.578652 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.578656 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.578661 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.578665 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.578669 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.578674 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.578678 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.578682 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.578687 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.578691 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.578695 | controller |       09:46:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.578700 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.578704 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.578708 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.578713 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.578719 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.578724 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.578728 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.578738 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.578743 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.578747 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.578752 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.578756 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.578760 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.578767 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.578771 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.578775 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.578780 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.578784 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.578788 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.578793 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.578797 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.578801 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.578806 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.578810 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.578814 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.578819 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.578823 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.578827 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.578832 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.578836 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.578840 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.578845 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.578849 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.578853 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.578858 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.578862 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.578866 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.578872 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.578877 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.578881 | controller |       09:46:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.578885 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.578890 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.578894 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.578899 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.578907 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.578913 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.578938 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.578945 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.578951 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.578975 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.578980 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.578985 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.578989 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.578994 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.578998 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.579002 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.579007 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.579011 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.579016 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.579020 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.579024 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.579029 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.579033 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.579037 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.579042 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.579046 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.579050 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.579055 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.579059 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.579066 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.579070 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.579074 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.579079 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.579083 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.579088 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.579094 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.579098 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.579103 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.579107 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.579111 | controller |       09:46:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.579116 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.579120 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.579124 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.579129 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.579133 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.579137 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.579142 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.579146 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.579150 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.579154 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.579166 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.579170 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.579175 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.579179 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.579183 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.579188 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.579192 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.579197 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.579201 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.579205 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.579212 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.579216 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.579220 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.579225 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.579229 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.579233 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.579238 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.579242 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.579246 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.579250 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.579255 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.579259 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.579264 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.579268 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.579272 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.579276 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.579281 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.579285 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.579289 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.579294 | controller |       09:46:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.579298 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.579302 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.579307 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.579311 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.579315 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.579320 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.579324 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.579328 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.579333 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.579337 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.579341 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.579346 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.579358 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.579365 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.579369 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.579375 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.579379 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.579384 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.579388 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.579393 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.579397 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.579401 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.579406 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.579410 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.579414 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.579419 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.579423 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.579427 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.579432 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.579436 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.579440 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.579445 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.579449 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.579453 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.579458 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.579462 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.579466 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.579471 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.579475 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.579479 | controller |       09:46:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.579484 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.579488 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.579492 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.579497 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.579504 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.579510 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.579514 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.579519 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.579523 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.579527 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.579532 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.579536 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.579540 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.579551 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.579556 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.579560 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.579565 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.579569 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.579573 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.579578 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.579582 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.579588 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.579593 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.579611 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.579616 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.579620 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.579625 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.579629 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.579633 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.579638 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.579642 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.579646 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.579651 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.579657 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.579661 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.579666 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.579670 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.579676 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.579681 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.579685 | controller |       09:46:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.579689 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.579694 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.579698 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.579702 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.579707 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.579711 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.579715 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.579720 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.579724 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.579728 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.579733 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.579737 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.579741 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.579746 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.579750 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.579762 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.579767 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.579771 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.579776 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.579780 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.579784 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.579789 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.579793 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.579798 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.579802 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.579806 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.579811 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.579815 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.579819 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.579825 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.579831 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.579836 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.579840 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.579845 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.579849 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.579853 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.579858 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.579862 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.579866 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.579871 | controller |       09:46:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.579875 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.579879 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.579884 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.579888 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.579892 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.579897 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.579901 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.579905 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.579910 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.579914 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.579918 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.579925 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.579929 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.579934 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.579938 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.579942 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.579953 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.579958 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.579962 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.579967 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.579973 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.579977 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.579982 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.579986 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.579991 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.579995 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.579999 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.580004 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.580008 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.580012 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.580017 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.580021 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.580025 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.580030 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.580036 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.580042 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.580048 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.580054 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.580058 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.580063 | controller |       09:46:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.580067 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.580071 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.580076 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.580080 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.580084 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.580089 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.580093 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.580097 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.580102 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.580106 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.580110 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.580115 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.580119 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.580134 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.580147 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.580152 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.580157 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.580169 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.580222 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.580227 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.580232 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.580237 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.580242 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.580255 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.580269 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.580273 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.580279 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.580284 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.580292 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.580297 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.580301 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.580305 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.580310 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.580314 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.580318 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.580323 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.580327 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.580331 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.580336 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.580340 | controller |       09:46:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.580344 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.580349 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.580353 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.580358 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.580365 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.580370 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.580374 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.580379 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.580383 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.580387 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.580392 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.580396 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.580400 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.580405 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.580409 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.580413 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.580418 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.580422 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.580426 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.580437 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.580442 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.580446 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.580450 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.580455 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.580459 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.580464 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.580468 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.580472 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.580477 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.580481 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.580485 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.580490 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.580494 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.580498 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.580503 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.580507 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.580511 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.580518 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.580522 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.580526 | controller |       09:46:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.580531 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.580535 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.580539 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.580544 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.580548 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.580552 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.580557 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.580561 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.580565 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.580570 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.580576 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.580580 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.580584 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.580589 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.580605 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.580612 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.580616 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.580621 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.580630 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.580635 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.580649 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.580654 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.580659 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.580663 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.580668 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.580672 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.580676 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.580681 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.580685 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.580691 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.580696 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.580700 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.580704 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.580709 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.580713 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.580718 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.580722 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.580726 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.580731 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.580735 | controller |       09:46:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.580740 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.580745 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.580751 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.580758 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.580764 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.580770 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.580776 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.580781 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.580787 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.580792 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.580796 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.580801 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.580805 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.580809 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.580818 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.580825 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.580830 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.580839 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.580939 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.580946 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.580954 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.580974 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.580979 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.580984 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.580988 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.580992 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.580997 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.581004 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.581009 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.581013 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.581017 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.581022 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.581026 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.581032 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.581041 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.581056 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.581061 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.581065 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.581070 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.581074 | controller |       09:46:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.581079 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.581085 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.581090 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.581095 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.581099 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.581108 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.581119 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.581124 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.581128 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.581133 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.581138 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.581142 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.581147 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.581158 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.581167 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.581172 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.581176 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.581181 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.581185 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.581189 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.581194 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.581198 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.581208 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.581212 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.581217 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.581221 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.581225 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.581230 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.581234 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.581238 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.581242 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.581247 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.581251 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.581256 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.581260 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.581264 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.581274 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.581278 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.581283 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.581287 | controller |       09:46:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.581292 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.581299 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.581304 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.581308 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.581314 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.581319 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.581323 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.581327 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.581332 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.581338 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.581342 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.581346 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.581351 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.581355 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.581359 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.581364 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.581368 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.581372 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.581377 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.581381 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.581385 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.581390 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.581394 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.581398 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.581409 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.581413 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.581418 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.581422 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.581427 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.581431 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.581437 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.581442 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.581446 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.581450 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.581455 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.581459 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.581463 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.581477 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.581483 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.581488 | controller |       09:46:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.581497 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.581501 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.581506 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.581511 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.581515 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.581519 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.581524 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.581529 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.581533 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.581539 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.581548 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.581560 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.581565 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.581570 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.581575 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.581581 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.581585 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.581590 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.581607 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.581615 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.581619 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.581623 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.581632 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.581638 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.581643 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.581650 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.581658 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.581664 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.581669 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.581679 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.581690 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.581694 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.581699 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.581704 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.581712 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.581725 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.581729 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.581733 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.581738 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.581744 | controller |       09:46:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.581752 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.581765 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.581769 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.581774 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.581779 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.581784 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.581790 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.581795 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.581800 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.581805 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.581811 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.581819 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.581865 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.581870 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.581874 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.581878 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.581883 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.581887 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.581891 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.581896 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.581905 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.581910 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.581914 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.581919 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.581923 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.581927 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.581932 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.581942 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.581946 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.581951 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.581955 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.581960 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.581964 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.581968 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.581972 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.581977 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.581981 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.581985 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.581990 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.581994 | controller |       09:46:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.581998 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.582003 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.582007 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.582011 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.582019 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.582023 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.582028 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.582032 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.582038 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.582043 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.582047 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.582051 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.582056 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.582062 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.582066 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.582070 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.582075 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.582079 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.582083 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.582088 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.582092 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.582097 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.582101 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.582105 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.582110 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.582114 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.582118 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.582122 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.582134 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.582138 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.582143 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.582147 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.582152 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.582156 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.582160 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.582165 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.582169 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.582173 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.582180 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.582184 | controller |       09:46:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.582188 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.582193 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.582197 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.582201 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.582208 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.582214 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.582220 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.582226 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.582232 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.582237 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.582241 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.582246 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.582250 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.582254 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.582259 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.582263 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.582267 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.582271 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.582276 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.582280 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.582284 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.582289 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.582293 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.582297 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.582304 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.582308 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.582312 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.582317 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.582321 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.582332 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.582343 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.582348 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.582352 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.582357 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.582361 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.582366 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.582370 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.582377 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.582382 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.582386 | controller |       09:46:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.582391 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.582395 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.582399 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.582403 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.582408 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.582412 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.582417 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.582423 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.582429 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.582436 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.582441 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.582447 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.582453 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.582457 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.582461 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.582468 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.582472 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.582477 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.582481 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.582485 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.582489 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.582494 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.582498 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.582502 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.582506 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.582511 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.582515 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.582519 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.582523 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.582530 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.582534 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.582545 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.582550 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.582554 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.582558 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.582562 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.582567 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.582571 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.582575 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.582579 | controller |       09:46:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.582584 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.582588 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.582592 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.582617 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.582622 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.582627 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.582631 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.582635 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.582639 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.582644 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.582648 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.582652 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.582656 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.582660 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.582665 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.582669 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.582673 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.582677 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.582682 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.582686 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.582692 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.582696 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.582701 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.582705 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.582709 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.582713 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.582717 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.582722 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.582726 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.582730 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.582734 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.582739 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.582749 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.582754 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.582758 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.582762 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.582767 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.582771 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.582775 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.582779 | controller |       09:46:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.582784 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.582788 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.582792 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.582796 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.582801 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.582805 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.582809 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.582813 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.582818 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.582822 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.582826 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.582830 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.582835 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.582841 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.582845 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.582849 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.582853 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.582858 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.582862 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.582866 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.582872 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.582877 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.582881 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.582887 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.582891 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.582895 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.582900 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.582904 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.582908 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.582913 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.582934 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.582940 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.582944 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.582949 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.582959 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.582964 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.582968 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.582972 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.582977 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.582984 | controller |       09:46:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.582989 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.582993 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.582997 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.583001 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.583008 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.583012 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.583016 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.583020 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.583025 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.583029 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.583033 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.583037 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.583042 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.583046 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.583050 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.583054 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.583058 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.583063 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.583067 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.583071 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.583075 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.583080 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.583084 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.583088 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.583092 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.583097 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.583101 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.583105 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.583109 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.583114 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.583118 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.583122 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.583126 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.583131 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.583135 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.583144 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.583149 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.583155 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.583159 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.583164 | controller |       09:46:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.583168 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.583172 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.583178 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.583183 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.583187 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.583191 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.583195 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.583200 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.583204 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.583208 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.583212 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.583217 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.583221 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.583225 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.583229 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.583234 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.583238 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.583242 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.583246 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.583251 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.583255 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.583259 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.583263 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.583268 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.583272 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.583276 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.583282 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.583288 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.583294 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.583302 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.583306 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.583310 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.583315 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.583319 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.583323 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.583327 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.583337 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.583342 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.583346 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.583351 | controller |       09:46:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.583355 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.583359 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.583363 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.583368 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.583374 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.583378 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.583383 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.583387 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.583391 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.583395 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.583400 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.583404 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.583408 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.583412 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.583417 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.583421 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.583425 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.583429 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.583434 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.583438 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.583444 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.583450 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.583454 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.583458 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.583462 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.583467 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.583471 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.583475 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.583479 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.583483 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.583488 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.583492 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.583496 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.583500 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.583505 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.583509 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.583513 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.583517 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.583527 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.583531 | controller |       09:46:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.583536 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.583540 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.583551 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.583555 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.583559 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.583564 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.583568 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.583572 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.583576 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.583581 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.583585 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.583589 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.583604 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.583613 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.583617 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.583621 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.583626 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.583630 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.583634 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.583638 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.583643 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.583647 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.583651 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.583656 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.583660 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.583664 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.583668 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.583672 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.583677 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.583681 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.583685 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.583689 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.583694 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.583698 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.583702 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.583706 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.583711 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.583715 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.583719 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.583729 | controller |       09:46:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.583734 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.583740 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.583745 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.583749 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.583755 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.583759 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.583764 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.583768 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.583772 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.583776 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.583780 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.583785 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.583789 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.583793 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.583797 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.583803 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.583808 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.583812 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.583816 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.583820 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.583826 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.583830 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.583835 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.583839 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.583843 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.583848 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.583852 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.583856 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.583860 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.583865 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.583869 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.583873 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.583877 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.583882 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.583886 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.583890 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.583895 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.583901 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.583905 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.583909 | controller |       09:46:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.583919 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.583923 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.583928 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.583932 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.583936 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.583941 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.583945 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.583949 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.583953 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.583958 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.583962 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.583966 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.583970 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.583975 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.583979 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.583983 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.583990 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.583995 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.583999 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.584003 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.584007 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.584012 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.584016 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.584020 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.584025 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.584029 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.584033 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.584037 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.584041 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.584049 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.584053 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.584057 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.584061 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.584066 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.584070 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.584074 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.584079 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.584083 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.584087 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.584091 | controller |       09:46:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.584096 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.584100 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.584109 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.584114 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.584118 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.584122 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.584127 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.584131 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.584135 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.584140 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.584144 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.584148 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.584152 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.584156 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.584161 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.584165 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.584169 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.584173 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.584178 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.584182 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.584188 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.584192 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.584196 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.584201 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.584205 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.584209 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.584213 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.584218 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.584222 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.584226 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.584232 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.584236 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.584241 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.584245 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.584249 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.584253 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.584258 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.584262 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.584266 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.584271 | controller |       09:46:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.584276 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.584281 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.584285 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.584295 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.584299 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.584305 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.584310 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.584314 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.584318 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.584322 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.584327 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.584331 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.584335 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.584341 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.584347 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.584353 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.584360 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.584366 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.584372 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.584378 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.584382 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.584387 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.584391 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.584395 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.584399 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.584404 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.584408 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.584412 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.584416 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.584421 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.584425 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.584429 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.584434 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.584438 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.584442 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.584446 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.584451 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.584455 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.584459 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.584463 | controller |       09:46:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.584468 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.584472 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.584476 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.584480 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.584487 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.584497 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.584502 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.584506 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.584510 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.584515 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.584519 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.584523 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.584527 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.584532 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.584538 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.584543 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.584547 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.584551 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.584555 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.584560 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.584564 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.584568 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.584572 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.584577 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.584581 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.584585 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.584589 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.584605 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.584611 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.584616 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.584620 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.584624 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.584628 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.584633 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.584637 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.584641 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.584645 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.584652 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.584656 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.584660 | controller |       09:46:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.584665 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:46:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.584669 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.584673 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.584677 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.584683 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.584688 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.584698 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.584703 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.584707 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.584711 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.584716 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.584720 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.584724 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.584728 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.584733 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.584737 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.584741 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.584746 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.584750 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.584754 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.584760 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.584764 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.584769 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.584773 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.584777 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.584782 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.584786 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.584790 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.584794 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.584800 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.584805 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.584809 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.584813 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.584817 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.584822 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.584826 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.584830 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.584834 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:46:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.584840 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.584845 | controller |       09:47:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.584849 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:47:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.584853 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.584858 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.584862 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.584866 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.584870 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.584875 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.584884 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.584889 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.584893 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.584897 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.584902 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.584906 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.584910 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.584914 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.584919 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.584923 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.584927 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.584932 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.584936 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.584942 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.584946 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.584950 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.584955 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.584959 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.584963 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.584967 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.584972 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.584976 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.584980 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.584985 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.584989 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.584993 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.584997 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.585002 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.585006 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.585011 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.585016 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:47:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.585020 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.585024 | controller |       09:47:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.585029 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:47:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.585033 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.585037 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.585041 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.585046 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.585050 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.585054 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.585058 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.585063 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.585072 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.585077 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.585081 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.585085 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.585095 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.585099 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.585105 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.585109 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.585113 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.585117 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.585122 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.585126 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.585130 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.585135 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.585139 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.585143 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.585147 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.585152 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.585156 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.585160 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.585164 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.585169 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.585173 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.585177 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.585181 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.585186 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.585190 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.585194 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.585198 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:47:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.585203 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.585207 | controller |       09:47:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.585211 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:47:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.585216 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.585220 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.585224 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.585230 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.585234 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.585238 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.585243 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.585247 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.585251 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.585261 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.585266 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.585270 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.585274 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.585278 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.585283 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.585287 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.585291 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.585296 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.585300 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.585304 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.585308 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.585313 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.585317 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.585321 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.585325 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.585330 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.585334 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.585338 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.585342 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.585347 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.585353 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.585357 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.585362 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.585366 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.585371 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.585376 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.585382 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:47:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.585386 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.585390 | controller |       09:47:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.585395 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:47:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.585399 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.585403 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.585408 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.585412 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.585416 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.585420 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.585426 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.585432 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.585438 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.585445 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.585450 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.585463 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.585468 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.585472 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.585476 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.585481 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.585485 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.585489 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.585493 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.585498 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.585502 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.585506 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.585510 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.585515 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.585519 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.585523 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.585527 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.585532 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.585538 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.585550 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.585554 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.585559 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.585563 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.585567 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.585571 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.585576 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.585580 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:47:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.585584 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.585589 | controller |       09:47:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.585603 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:47:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.585611 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.585615 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.585619 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.585623 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.585628 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.585632 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.585636 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.585640 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.585645 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.585649 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.585653 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.585659 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.585670 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.585674 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.585678 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.585683 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.585687 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.585691 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.585696 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.585704 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.585709 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.585713 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.585717 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.585721 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.585725 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.585730 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.585734 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.585738 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.585743 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.585747 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.585751 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.585755 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.585760 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.585764 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.585768 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.585772 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.585777 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:47:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.585781 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.585785 | controller |       09:47:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.585790 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:47:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.585794 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.585798 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.585802 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.585807 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.585811 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.585815 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.585819 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.585824 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.585828 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.585832 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.585836 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.585841 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.585847 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.585851 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.585861 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.585865 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.585869 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.585874 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.585878 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.585882 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.585887 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.585891 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.585895 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.585900 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.585904 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.585908 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.585912 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.585917 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.585921 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.585925 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.585929 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.585935 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.585940 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.585944 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.585948 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.585953 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.585957 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:47:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.585961 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.585966 | controller |       09:47:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.585970 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:47:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.585974 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.585978 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.585983 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.585990 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.585995 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.585999 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.586003 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.586008 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.586012 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.586016 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.586021 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.586025 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.586029 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.586033 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.586038 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.586047 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.586051 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.586056 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.586060 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.586064 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.586069 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.586073 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.586077 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.586081 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.586086 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.586090 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.586094 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.586098 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.586103 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.586107 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.586111 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.586115 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.586120 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.586124 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.586128 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.586133 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.586140 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:47:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.586144 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.586149 | controller |       09:47:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.586153 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:47:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.586157 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.586162 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.586166 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.586170 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.586174 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.586179 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.586183 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.586187 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.586191 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.586196 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.586202 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.586206 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.586210 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.586215 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.586219 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.586223 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.586232 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.586237 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.586241 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.586246 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.586250 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.586254 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.586258 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.586263 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.586267 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.586271 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.586275 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.586280 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.586286 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.586290 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.586294 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.586299 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.586303 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.586307 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.586312 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.586316 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.586320 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:47:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.586324 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.586329 | controller |       09:47:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.586333 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:47:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.586337 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.586341 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.586346 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.586350 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.586354 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.586359 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.586363 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.586367 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.586371 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.586376 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.586380 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.586384 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.586389 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.586393 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.586399 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.586403 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.586407 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.586417 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.586421 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.586427 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.586432 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.586436 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.586440 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.586444 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.586449 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.586453 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.586457 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.586463 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.586467 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.586472 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.586476 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.586480 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.586485 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.586489 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.586493 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.586497 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.586502 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:47:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.586510 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.586516 | controller |       09:47:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.586522 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:47:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.586528 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.586534 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.586540 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.586544 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.586549 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.586553 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.586557 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.586561 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.586566 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.586570 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.586574 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.586578 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.586585 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.586589 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.586604 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.586610 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.586615 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.586619 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.586623 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.586636 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.586641 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.586646 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.586650 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.586654 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.586659 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.586663 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.586667 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.586671 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.586675 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.586680 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.586684 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.586688 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.586692 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.586697 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.586701 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.586705 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.586709 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:47:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.586714 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.586718 | controller |       09:47:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.586722 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:47:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.586726 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.586731 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.586735 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.586741 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.586745 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.586750 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.586754 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.586758 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.586762 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.586768 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.586772 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.586777 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.586781 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.586785 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.586789 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.586794 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.586798 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.586802 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.586806 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.586811 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.586820 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.586825 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.586829 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.586833 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.586837 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.586842 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.586846 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.586850 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.586854 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.586861 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.586865 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.586869 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.586873 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.586878 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.586882 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.586886 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.586892 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:47:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.586896 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.586901 | controller |       09:47:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.586905 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:47:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.586909 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.586913 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.586918 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.586937 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.586943 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.586947 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.586951 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.586955 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.586960 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.586964 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.586968 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.586972 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.586976 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.586981 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.586985 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.586989 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.586993 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.586998 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.587002 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.587006 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.587010 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.587020 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.587025 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.587029 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.587033 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.587038 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.587044 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.587049 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.587055 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.587059 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.587063 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.587068 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.587072 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.587076 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.587080 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.587085 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.587089 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:47:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.587093 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.587098 | controller |       09:47:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.587102 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:47:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.587106 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.587110 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.587115 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.587119 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.587123 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.587127 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.587132 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.587136 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.587140 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.587144 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.587149 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.587153 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.587157 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.587161 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.587165 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.587170 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.587174 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.587178 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.587182 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.587190 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.587194 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.587198 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.587202 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.587212 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.587217 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.587221 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.587225 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.587229 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.587234 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.587238 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.587242 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.587246 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.587251 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.587255 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.587259 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.587263 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.587268 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:47:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.587272 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.587276 | controller |       09:47:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.587280 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:47:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.587285 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.587289 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.587293 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.587299 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.587303 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.587308 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.587312 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.587316 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.587322 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.587326 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.587331 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.587336 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.587343 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.587347 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.587353 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.587358 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.587363 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.587367 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.587371 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.587375 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.587380 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.587384 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.587388 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.587392 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.587402 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.587406 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.587411 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.587415 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.587419 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.587423 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.587428 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.587432 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.587436 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.587441 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.587445 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.587449 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.587453 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:47:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.587458 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.587462 | controller |       09:47:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.587466 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:47:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.587470 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.587475 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.587479 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.587486 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.587490 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.587494 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.587498 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.587503 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.587507 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.587511 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.587515 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.587520 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.587524 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.587528 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.587532 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.587537 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.587541 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.587545 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.587549 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.587555 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.587560 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.587564 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.587568 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.587572 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.587578 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.587583 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.587609 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.587619 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.587625 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.587630 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.587636 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.587641 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.587647 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.587652 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.587658 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.587663 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.587672 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:47:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.587678 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.587683 | controller |       09:47:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.587689 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:47:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.587694 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.587700 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.587706 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.587712 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.587717 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.587723 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.587729 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.587735 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.587741 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.587747 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.587753 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.587759 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.587765 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.587771 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.587781 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.587788 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.587794 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.587800 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.587806 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.587812 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.587819 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.587825 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.587834 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.587840 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.587846 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.587852 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.587858 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.587877 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.587887 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.587894 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.587900 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.587906 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.587912 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.587918 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.587924 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.587930 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.587935 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:47:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.587941 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.587947 | controller |       09:47:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.587953 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:47:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.587959 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.587965 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.587970 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.587976 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.587982 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.587988 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.587994 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.588004 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.588010 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.588016 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.588022 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.588028 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.588034 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.588040 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.588047 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.588052 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.588059 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.588065 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.588071 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.588080 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.588086 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.588092 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.588098 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.588104 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.588110 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.588116 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.588122 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.588128 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.588134 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.588150 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.588156 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.588162 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.588168 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.588174 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.588180 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.588186 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.588192 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:47:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.588198 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.588204 | controller |       09:47:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.588210 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:47:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.588215 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:56:26.588221 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:56:26.588228 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:56:26.588234 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:56:26.588240 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:56:26.588245 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:56:26.588251 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:56:26.588257 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:56:26.588263 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:56:26.588269 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:56:26.588275 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:56:26.588281 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.588290 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.588296 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:56:26.588301 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:56:26.588307 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:56:26.588313 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:56:26.588319 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:56:26.588325 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.588332 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:56:26.588337 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:56:26.588343 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:56:26.588349 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:56:26.588358 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:56:26.588364 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:56:26.588370 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:56:26.588376 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:56:26.588381 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:56:26.588387 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:56:26.588396 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:56:26.588412 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.588418 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:56:26.588423 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:56:26.588429 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.588434 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:56:26.588440 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:56:26.588445 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:47:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.588451 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:56:26.588457 | controller |       09:47:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:56:26.588463 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:47:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:56:26.588469 | controller |       | test step failed 2-deploy-appcred\n    case.go:396: failed in step 2-deploy-appcred\n
2026-03-14 09:56:26.588475 | controller |       \   case.go:398: command \"NS=\\\"${NAMESPACE}\\\"\\\\n echo \\\"Waiting for KeystoneApplicationCredential...\"
2026-03-14 09:56:26.588481 | controller |       failed, exit status 1\n    logger.go:42: 09:47:22 | appcred-tests | skipping kubernetes
2026-03-14 09:56:26.588491 | controller |       event logging\n=== CONT  kuttl/harness/watcher-notification\n    logger.go:42: 09:47:22
2026-03-14 09:56:26.588497 | controller |       | watcher-notification | Skipping creation of user-supplied namespace: watcher-kuttl-default\n
2026-03-14 09:56:26.588504 | controller |       \   logger.go:42: 09:47:22 | watcher-notification/0-cleanup-watcher | starting test
2026-03-14 09:56:26.588510 | controller |       step 0-cleanup-watcher\n    logger.go:42: 09:47:22 | watcher-notification/0-cleanup-watcher
2026-03-14 09:56:26.588516 | controller |       | test step completed 0-cleanup-watcher\n    logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.588522 | controller |       | starting test step 1-deploy-with-notification\n    logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.588529 | controller |       | Watcher:watcher-kuttl-default/watcher-kuttl created\n    logger.go:42: 09:47:22
2026-03-14 09:56:26.588535 | controller |       | watcher-notification/1-deploy-with-notification | running command: [sh -c set
2026-03-14 09:56:26.588541 | controller |       -euxo pipefail\n        oc project watcher-kuttl-default\n        APIPOD=$(oc get
2026-03-14 09:56:26.588547 | controller |       pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep
2026-03-14 09:56:26.588553 | controller |       -v ^$|head -1)\n        if [ -n \"${APIPOD}\" ]; then\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.588560 | controller |       -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep
2026-03-14 09:56:26.588566 | controller |       -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.588572 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.588578 | controller |       09:47:22 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.588585 | controller |       \   logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification | Now
2026-03-14 09:56:26.588591 | controller |       using project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.588618 | controller |       \   logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.588625 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.588631 | controller |       \   logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.588638 | controller |       head -1\n    logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.588644 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.588650 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.588656 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.588663 | controller |       | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n
2026-03-14 09:56:26.588669 | controller |       \   logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification | +++
2026-03-14 09:56:26.588675 | controller |       oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.588682 | controller |       \   logger.go:42: 09:47:23 | watcher-notification/1-deploy-with-notification | error:
2026-03-14 09:56:26.588700 | controller |       Internal error occurred: error executing command in container: container is not
2026-03-14 09:56:26.588707 | controller |       created or running\n    logger.go:42: 09:47:23 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.588714 | controller |       | ++ echo\n    logger.go:42: 09:47:23 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.588720 | controller |       | + '[' 0 == 1 ']'\n    logger.go:42: 09:47:24 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.588727 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.588737 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.588743 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.588749 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.588755 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.588762 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.588768 | controller |       09:47:24 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.588778 | controller |       \   logger.go:42: 09:47:25 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:56:26.588784 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.588791 | controller |       \   logger.go:42: 09:47:25 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.588797 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.588803 | controller |       \   logger.go:42: 09:47:25 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.588809 | controller |       head -1\n    logger.go:42: 09:47:25 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.588815 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:47:25 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.588821 | controller |       | + APIPOD=\n    logger.go:42: 09:47:26 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.588827 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.588833 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.588839 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.588845 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.588851 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.588857 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.588864 | controller |       09:47:26 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.588870 | controller |       \   logger.go:42: 09:47:26 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:56:26.588876 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.588882 | controller |       \   logger.go:42: 09:47:26 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.588888 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.588895 | controller |       \   logger.go:42: 09:47:26 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.588901 | controller |       head -1\n    logger.go:42: 09:47:26 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.588907 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:47:26 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.588913 | controller |       | + APIPOD=\n    logger.go:42: 09:47:27 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.588919 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.588925 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.588932 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.588938 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.588947 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.588953 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.588962 | controller |       09:47:27 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.588969 | controller |       \   logger.go:42: 09:47:27 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:56:26.588975 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.588992 | controller |       \   logger.go:42: 09:47:27 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589000 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.589006 | controller |       \   logger.go:42: 09:47:27 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589013 | controller |       head -1\n    logger.go:42: 09:47:27 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589019 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:47:28 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589025 | controller |       | + APIPOD=\n    logger.go:42: 09:47:29 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589035 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.589041 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.589047 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.589054 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.589060 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.589066 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.589073 | controller |       09:47:29 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.589079 | controller |       \   logger.go:42: 09:47:29 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:56:26.589085 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.589091 | controller |       \   logger.go:42: 09:47:29 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589098 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.589104 | controller |       \   logger.go:42: 09:47:29 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589113 | controller |       head -1\n    logger.go:42: 09:47:29 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589120 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:47:29 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589126 | controller |       | + APIPOD=\n    logger.go:42: 09:47:30 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589132 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.589138 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.589145 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.589151 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.589157 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.589163 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.589172 | controller |       09:47:30 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.589179 | controller |       \   logger.go:42: 09:47:30 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:56:26.589185 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.589191 | controller |       \   logger.go:42: 09:47:30 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589197 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.589203 | controller |       \   logger.go:42: 09:47:30 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589210 | controller |       head -1\n    logger.go:42: 09:47:30 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589216 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:47:30 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589222 | controller |       | + APIPOD=\n    logger.go:42: 09:47:31 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589227 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.589233 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.589238 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.589244 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.589250 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.589257 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.589274 | controller |       09:47:31 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.589280 | controller |       \   logger.go:42: 09:47:31 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:56:26.589284 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.589288 | controller |       \   logger.go:42: 09:47:31 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589293 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.589297 | controller |       \   logger.go:42: 09:47:31 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589302 | controller |       head -1\n    logger.go:42: 09:47:31 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589306 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:47:32 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589310 | controller |       | + APIPOD=\n    logger.go:42: 09:47:33 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589315 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.589319 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.589323 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.589328 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.589332 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.589336 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.589341 | controller |       09:47:33 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.589345 | controller |       \   logger.go:42: 09:47:33 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:56:26.589350 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.589357 | controller |       \   logger.go:42: 09:47:33 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589361 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.589365 | controller |       \   logger.go:42: 09:47:33 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589370 | controller |       head -1\n    logger.go:42: 09:47:33 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589374 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:47:33 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589378 | controller |       | + APIPOD=\n    logger.go:42: 09:47:34 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589383 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.589387 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.589391 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.589396 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.589400 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.589407 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.589411 | controller |       09:47:34 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.589416 | controller |       \   logger.go:42: 09:47:34 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:56:26.589420 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.589425 | controller |       \   logger.go:42: 09:47:34 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589429 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.589433 | controller |       \   logger.go:42: 09:47:34 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589438 | controller |       grep -v '^$'\n    logger.go:42: 09:47:34 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589442 | controller |       | ++ head -1\n    logger.go:42: 09:47:34 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589446 | controller |       | + APIPOD=\n    logger.go:42: 09:47:35 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589451 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.589455 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.589459 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.589464 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.589475 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.589480 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.589486 | controller |       09:47:35 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.589491 | controller |       \   logger.go:42: 09:47:35 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:56:26.589495 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.589499 | controller |       \   logger.go:42: 09:47:35 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589504 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.589508 | controller |       \   logger.go:42: 09:47:35 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589514 | controller |       head -1\n    logger.go:42: 09:47:35 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589519 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:47:36 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589523 | controller |       | + APIPOD=\n    logger.go:42: 09:47:37 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589527 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.589532 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.589536 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.589540 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.589545 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.589549 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.589553 | controller |       09:47:37 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.589558 | controller |       \   logger.go:42: 09:47:37 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:56:26.589562 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.589566 | controller |       \   logger.go:42: 09:47:37 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589571 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.589575 | controller |       \   logger.go:42: 09:47:37 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589579 | controller |       head -1\n    logger.go:42: 09:47:37 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589584 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:47:37 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589588 | controller |       | + APIPOD=\n    logger.go:42: 09:47:38 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589592 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.589613 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.589618 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.589622 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.589627 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.589631 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.589635 | controller |       09:47:38 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.589640 | controller |       \   logger.go:42: 09:47:38 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:56:26.589644 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.589648 | controller |       \   logger.go:42: 09:47:38 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589653 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.589657 | controller |       \   logger.go:42: 09:47:38 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589663 | controller |       head -1\n    logger.go:42: 09:47:38 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589667 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:47:38 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589672 | controller |       | + APIPOD=\n    logger.go:42: 09:47:39 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589678 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.589682 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.589694 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.589699 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.589703 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.589708 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.589712 | controller |       09:47:39 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.589716 | controller |       \   logger.go:42: 09:47:40 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:56:26.589721 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.589725 | controller |       \   logger.go:42: 09:47:40 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589729 | controller |       grep -v '^$'\n    logger.go:42: 09:47:40 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589734 | controller |       | ++ head -1\n    logger.go:42: 09:47:40 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589738 | controller |       | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.589742 | controller |       \   logger.go:42: 09:47:40 | watcher-notification/1-deploy-with-notification | +
2026-03-14 09:56:26.589747 | controller |       APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:47:40 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589751 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:47:40 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589756 | controller |       | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n
2026-03-14 09:56:26.589760 | controller |       \   logger.go:42: 09:47:40 | watcher-notification/1-deploy-with-notification | +++
2026-03-14 09:56:26.589764 | controller |       oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.589769 | controller |       \   logger.go:42: 09:47:40 | watcher-notification/1-deploy-with-notification | error:
2026-03-14 09:56:26.589773 | controller |       unable to upgrade connection: container not found (\"watcher-api\")\n    logger.go:42:
2026-03-14 09:56:26.589777 | controller |       09:47:40 | watcher-notification/1-deploy-with-notification | ++ echo\n    logger.go:42:
2026-03-14 09:56:26.589781 | controller |       09:47:40 | watcher-notification/1-deploy-with-notification | + '[' 0 == 1 ']'\n
2026-03-14 09:56:26.589786 | controller |       \   logger.go:42: 09:47:41 | watcher-notification/1-deploy-with-notification | running
2026-03-14 09:56:26.589790 | controller |       command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n        APIPOD=$(oc
2026-03-14 09:56:26.589794 | controller |       get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep
2026-03-14 09:56:26.589799 | controller |       -v ^$|head -1)\n        if [ -n \"${APIPOD}\" ]; then\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.589803 | controller |       -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep
2026-03-14 09:56:26.589807 | controller |       -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.589812 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.589816 | controller |       09:47:41 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.589820 | controller |       \   logger.go:42: 09:47:41 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:56:26.589828 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.589833 | controller |       \   logger.go:42: 09:47:41 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589837 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.589841 | controller |       \   logger.go:42: 09:47:41 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589846 | controller |       grep -v '^$'\n    logger.go:42: 09:47:41 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589850 | controller |       | ++ head -1\n    logger.go:42: 09:47:41 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589854 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:47:41 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589859 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:47:41 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.589863 | controller |       | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n
2026-03-14 09:56:26.589867 | controller |       \   logger.go:42: 09:47:41 | watcher-notification/1-deploy-with-notification | +++
2026-03-14 09:56:26.589874 | controller |       oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.589878 | controller |       \   logger.go:42: 09:47:42 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.589890 | controller |       echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.589895 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.589899 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.589903 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_b745:b0f48e43119d15c3491ae912affe1d4b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.589908 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.589913 | controller |       driver = messagingv2 transport_url = 'rabbit://default_user_KcunVqmb0RRrZ6hA1Rz:BEDiv_wgPdFx5S0HWqhob2_UqEzAdLGa@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.589917 | controller |       '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.589921 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.589926 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:56:26.589930 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:56:26.589935 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.589939 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.589943 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.589948 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.589952 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.589956 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.589961 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.589967 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.589974 | controller |       tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:56:26.589978 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]'
2026-03-14 09:56:26.589983 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.589987 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.589991 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.589996 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.590000 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.590004 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.590010 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.590015 | controller |       period = 900\n    logger.go:42: 09:47:42 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590019 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590023 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.590028 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.590032 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.590036 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.590041 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.590045 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.590049 | controller |       09:47:43 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.590054 | controller |       \   logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:56:26.590058 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.590062 | controller |       \   logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.590074 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.590078 | controller |       \   logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.590083 | controller |       head -1\n    logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590087 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590091 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590096 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590100 | controller |       | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n
2026-03-14 09:56:26.590105 | controller |       \   logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification | +++
2026-03-14 09:56:26.590109 | controller |       oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.590113 | controller |       \   logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.590118 | controller |       echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.590124 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.590128 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.590133 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_b745:b0f48e43119d15c3491ae912affe1d4b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.590137 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.590142 | controller |       driver = messagingv2 transport_url = 'rabbit://default_user_KcunVqmb0RRrZ6hA1Rz:BEDiv_wgPdFx5S0HWqhob2_UqEzAdLGa@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.590146 | controller |       '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.590150 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.590155 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:56:26.590159 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:56:26.590163 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.590168 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.590172 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.590177 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.590181 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.590185 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.590189 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.590194 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.590198 | controller |       tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:56:26.590202 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]'
2026-03-14 09:56:26.590207 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.590211 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.590216 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.590220 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.590224 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.590229 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.590233 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.590237 | controller |       period = 900\n    logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590248 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:47:44 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590252 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.590259 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.590264 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.590268 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.590273 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.590279 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.590283 | controller |       09:47:44 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.590288 | controller |       \   logger.go:42: 09:47:44 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:56:26.590292 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.590296 | controller |       \   logger.go:42: 09:47:44 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.590301 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.590305 | controller |       \   logger.go:42: 09:47:44 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.590309 | controller |       head -1\n    logger.go:42: 09:47:44 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590314 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:47:45 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590318 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:47:45 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590322 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:47:45 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590328 | controller |       | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n
2026-03-14 09:56:26.590334 | controller |       \   logger.go:42: 09:47:45 | watcher-notification/1-deploy-with-notification | +++
2026-03-14 09:56:26.590340 | controller |       oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.590344 | controller |       \   logger.go:42: 09:47:45 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.590349 | controller |       echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.590354 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.590360 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.590364 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_b745:b0f48e43119d15c3491ae912affe1d4b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.590370 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.590374 | controller |       driver = messagingv2 transport_url = 'rabbit://default_user_KcunVqmb0RRrZ6hA1Rz:BEDiv_wgPdFx5S0HWqhob2_UqEzAdLGa@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.590379 | controller |       '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.590383 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.590387 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:56:26.590392 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:56:26.590396 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.590402 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.590407 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.590411 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.590415 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.590420 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.590424 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.590428 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.590438 | controller |       tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:56:26.590443 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]'
2026-03-14 09:56:26.590447 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.590452 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.590456 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.590461 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.590465 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.590469 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.590474 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.590478 | controller |       period = 900\n    logger.go:42: 09:47:45 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590482 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590487 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.590491 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.590495 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.590500 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.590504 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.590508 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.590513 | controller |       09:47:46 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.590517 | controller |       \   logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:56:26.590521 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.590526 | controller |       \   logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.590530 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.590534 | controller |       \   logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.590538 | controller |       head -1\n    logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590543 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590549 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590553 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590557 | controller |       | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n
2026-03-14 09:56:26.590562 | controller |       \   logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification | +++
2026-03-14 09:56:26.590566 | controller |       oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.590570 | controller |       \   logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.590575 | controller |       echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.590579 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.590585 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.590590 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_b745:b0f48e43119d15c3491ae912affe1d4b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.590605 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.590612 | controller |       driver = messagingv2 transport_url = 'rabbit://default_user_KcunVqmb0RRrZ6hA1Rz:BEDiv_wgPdFx5S0HWqhob2_UqEzAdLGa@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.590617 | controller |       '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.590629 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.590633 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:56:26.590638 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:56:26.590642 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.590647 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.590651 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.590656 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.590660 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.590664 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.590669 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.590673 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.590677 | controller |       tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:56:26.590682 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]'
2026-03-14 09:56:26.590686 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.590690 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.590695 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.590701 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.590708 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.590712 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.590717 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.590721 | controller |       period = 900\n    logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590726 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590730 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.590734 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.590739 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.590743 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.590747 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.590752 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.590756 | controller |       09:47:48 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.590760 | controller |       \   logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:56:26.590764 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.590769 | controller |       \   logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.590773 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.590777 | controller |       \   logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.590782 | controller |       head -1\n    logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590786 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590790 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590795 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590799 | controller |       | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n
2026-03-14 09:56:26.590803 | controller |       \   logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification | +++
2026-03-14 09:56:26.590808 | controller |       oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.590817 | controller |       \   logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.590822 | controller |       echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.590826 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.590831 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.590835 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_b745:b0f48e43119d15c3491ae912affe1d4b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.590840 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.590849 | controller |       driver = messagingv2 transport_url = 'rabbit://default_user_KcunVqmb0RRrZ6hA1Rz:BEDiv_wgPdFx5S0HWqhob2_UqEzAdLGa@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.590853 | controller |       '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.590857 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.590862 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:56:26.590866 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:56:26.590870 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.590877 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.590881 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.590885 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.590890 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.590894 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.590898 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.590903 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.590907 | controller |       tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:56:26.590913 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]'
2026-03-14 09:56:26.590917 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.590922 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.590939 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.590943 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.590948 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.590952 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.590956 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.590961 | controller |       period = 900\n    logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590965 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:47:49 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.590969 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.590974 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.590978 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.590982 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.590986 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.590991 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.590995 | controller |       09:47:49 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.591001 | controller |       \   logger.go:42: 09:47:49 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:56:26.591011 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.591016 | controller |       \   logger.go:42: 09:47:49 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.591020 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.591024 | controller |       \   logger.go:42: 09:47:49 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.591029 | controller |       head -1\n    logger.go:42: 09:47:49 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.591033 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:47:49 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.591037 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:47:49 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.591041 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:47:49 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.591046 | controller |       | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n
2026-03-14 09:56:26.591050 | controller |       \   logger.go:42: 09:47:49 | watcher-notification/1-deploy-with-notification | +++
2026-03-14 09:56:26.591054 | controller |       oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.591058 | controller |       \   logger.go:42: 09:47:50 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.591063 | controller |       echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.591067 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.591071 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.591075 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_b745:b0f48e43119d15c3491ae912affe1d4b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.591080 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.591084 | controller |       driver = messagingv2 transport_url = 'rabbit://default_user_KcunVqmb0RRrZ6hA1Rz:BEDiv_wgPdFx5S0HWqhob2_UqEzAdLGa@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.591088 | controller |       '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.591092 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.591097 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:56:26.591101 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:56:26.591105 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.591110 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.591114 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.591118 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.591122 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.591127 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.591131 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.591137 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.591141 | controller |       tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:56:26.591145 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]'
2026-03-14 09:56:26.591150 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.591154 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.591158 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.591162 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.591167 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.591171 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.591180 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.591185 | controller |       period = 900\n    logger.go:42: 09:47:50 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.591189 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.591193 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.591198 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.591202 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.591206 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.591211 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.591215 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.591219 | controller |       09:47:51 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.591225 | controller |       \   logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:56:26.591230 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.591234 | controller |       \   logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.591238 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.591243 | controller |       \   logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.591247 | controller |       grep -v '^$'\n    logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.591251 | controller |       | ++ head -1\n    logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.591255 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.591259 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.591264 | controller |       | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n
2026-03-14 09:56:26.591268 | controller |       \   logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification | +++
2026-03-14 09:56:26.591274 | controller |       oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.591278 | controller |       \   logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.591284 | controller |       echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.591288 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.591293 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.591297 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_b745:b0f48e43119d15c3491ae912affe1d4b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.591301 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.591306 | controller |       driver = messagingv2 transport_url = 'rabbit://default_user_KcunVqmb0RRrZ6hA1Rz:BEDiv_wgPdFx5S0HWqhob2_UqEzAdLGa@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.591310 | controller |       '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.591314 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.591318 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:56:26.591323 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:56:26.591329 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.591333 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.591337 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.591342 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.591346 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.591350 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.591360 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.591364 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.591369 | controller |       tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:56:26.591373 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]'
2026-03-14 09:56:26.591377 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.591383 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.591388 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.591393 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.591398 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.591402 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.591406 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.591411 | controller |       period = 900\n    logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.591415 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:47:52 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.591421 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.591425 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.591429 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.591434 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.591438 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:56:26.591442 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:56:26.591446 | controller |       09:47:52 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:56:26.591451 | controller |       \   logger.go:42: 09:47:52 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:56:26.591455 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.591459 | controller |       \   logger.go:42: 09:47:53 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.591463 | controller |       grep -v '^$'\n    logger.go:42: 09:47:53 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.591468 | controller |       | ++ head -1\n    logger.go:42: 09:47:53 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.591472 | controller |       | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.591476 | controller |       \   logger.go:42: 09:47:53 | watcher-notification/1-deploy-with-notification | +
2026-03-14 09:56:26.591480 | controller |       APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:47:53 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.591485 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:47:53 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.591489 | controller |       | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n
2026-03-14 09:56:26.591493 | controller |       \   logger.go:42: 09:47:53 | watcher-notification/1-deploy-with-notification | +++
2026-03-14 09:56:26.591497 | controller |       oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.591502 | controller |       \   logger.go:42: 09:47:53 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:56:26.591506 | controller |       echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.591510 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.591514 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.591519 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_b745:b0f48e43119d15c3491ae912affe1d4b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.591523 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.591532 | controller |       driver = messagingv2 transport_url = 'rabbit://default_user_KcunVqmb0RRrZ6hA1Rz:BEDiv_wgPdFx5S0HWqhob2_UqEzAdLGa@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.591537 | controller |       '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.591541 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.591546 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:56:26.591550 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:56:26.591557 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.591561 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.591567 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.591571 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.591575 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.591580 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.591584 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.591589 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.591607 | controller |       tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:56:26.591617 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]'
2026-03-14 09:56:26.591623 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.591629 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.591634 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.591638 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.591643 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.591647 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.591651 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.591656 | controller |       period = 900\n    logger.go:42: 09:47:53 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.591660 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:47:53 | watcher-notification/1-deploy-with-notification
2026-03-14 09:56:26.591664 | controller |       | test step completed 1-deploy-with-notification\n    logger.go:42: 09:47:53 | watcher-notification/2-cleanup-watcher
2026-03-14 09:56:26.591668 | controller |       | starting test step 2-cleanup-watcher\n    logger.go:42: 09:47:59 | watcher-notification/2-cleanup-watcher
2026-03-14 09:56:26.591673 | controller |       | test step completed 2-cleanup-watcher\n    logger.go:42: 09:47:59 | watcher-notification
2026-03-14 09:56:26.591677 | controller |       | skipping kubernetes event logging\n=== CONT  kuttl/harness/watcher-topology\n
2026-03-14 09:56:26.591681 | controller |       \   logger.go:42: 09:47:59 | watcher-topology | Skipping creation of user-supplied
2026-03-14 09:56:26.591686 | controller |       namespace: watcher-kuttl-default\n    logger.go:42: 09:47:59 | watcher-topology/0-cleanup-watcher
2026-03-14 09:56:26.591690 | controller |       | starting test step 0-cleanup-watcher\n    logger.go:42: 09:47:59 | watcher-topology/0-cleanup-watcher
2026-03-14 09:56:26.591694 | controller |       | test step completed 0-cleanup-watcher\n    logger.go:42: 09:47:59 | watcher-topology/1-deploy-with-topology
2026-03-14 09:56:26.591698 | controller |       | starting test step 1-deploy-with-topology\n    logger.go:42: 09:47:59 | watcher-topology/1-deploy-with-topology
2026-03-14 09:56:26.591703 | controller |       | Topology:watcher-kuttl-default/watcher-api created\n    logger.go:42: 09:47:59
2026-03-14 09:56:26.591707 | controller |       | watcher-topology/1-deploy-with-topology | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:56:26.591711 | controller |       created\n    logger.go:42: 09:48:24 | watcher-topology/1-deploy-with-topology |
2026-03-14 09:56:26.591716 | controller |       test step completed 1-deploy-with-topology\n    logger.go:42: 09:48:24 | watcher-topology/2-cleanup-watcher
2026-03-14 09:56:26.591722 | controller |       | starting test step 2-cleanup-watcher\n    logger.go:42: 09:48:24 | watcher-topology/2-cleanup-watcher
2026-03-14 09:56:26.591727 | controller |       | running command: [sh -c set -ex\n        oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.591731 | controller |       -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0
2026-03-14 09:56:26.591742 | controller |       ]\n        ]\n    logger.go:42: 09:48:24 | watcher-topology/2-cleanup-watcher |
2026-03-14 09:56:26.591747 | controller |       + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.591751 | controller |       value -c Name -c Type\n    logger.go:42: 09:48:24 | watcher-topology/2-cleanup-watcher
2026-03-14 09:56:26.591756 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:48:27 | watcher-topology/2-cleanup-watcher
2026-03-14 09:56:26.591760 | controller |       | + '[' 1 == 0 ']'\n    logger.go:42: 09:48:28 | watcher-topology/2-cleanup-watcher
2026-03-14 09:56:26.591764 | controller |       | running command: [sh -c set -ex\n        oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.591768 | controller |       -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0
2026-03-14 09:56:26.591773 | controller |       ]\n        ]\n    logger.go:42: 09:48:28 | watcher-topology/2-cleanup-watcher |
2026-03-14 09:56:26.591777 | controller |       + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.591781 | controller |       value -c Name -c Type\n    logger.go:42: 09:48:28 | watcher-topology/2-cleanup-watcher
2026-03-14 09:56:26.591786 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:48:30 | watcher-topology/2-cleanup-watcher
2026-03-14 09:56:26.591790 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:48:31 | watcher-topology/2-cleanup-watcher
2026-03-14 09:56:26.591794 | controller |       | running command: [sh -c set -ex\n        oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.591798 | controller |       -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0
2026-03-14 09:56:26.591803 | controller |       ]\n        ]\n    logger.go:42: 09:48:31 | watcher-topology/2-cleanup-watcher |
2026-03-14 09:56:26.591807 | controller |       + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.591811 | controller |       value -c Name -c Type\n    logger.go:42: 09:48:31 | watcher-topology/2-cleanup-watcher
2026-03-14 09:56:26.591815 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:48:33 | watcher-topology/2-cleanup-watcher
2026-03-14 09:56:26.591822 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:48:34 | watcher-topology/2-cleanup-watcher
2026-03-14 09:56:26.591826 | controller |       | running command: [sh -c set -ex\n        oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.591830 | controller |       -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0
2026-03-14 09:56:26.591835 | controller |       ]\n        ]\n    logger.go:42: 09:48:34 | watcher-topology/2-cleanup-watcher |
2026-03-14 09:56:26.591839 | controller |       + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.591843 | controller |       value -c Name -c Type\n    logger.go:42: 09:48:34 | watcher-topology/2-cleanup-watcher
2026-03-14 09:56:26.591847 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:48:37 | watcher-topology/2-cleanup-watcher
2026-03-14 09:56:26.591851 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:48:37 | watcher-topology/2-cleanup-watcher
2026-03-14 09:56:26.591856 | controller |       | test step completed 2-cleanup-watcher\n    logger.go:42: 09:48:37 | watcher-topology
2026-03-14 09:56:26.591860 | controller |       | skipping kubernetes event logging\n=== CONT  kuttl/harness/watcher-tls-certs-change\n
2026-03-14 09:56:26.591864 | controller |       \   logger.go:42: 09:48:37 | watcher-tls-certs-change | Skipping creation of user-supplied
2026-03-14 09:56:26.591868 | controller |       namespace: watcher-kuttl-default\n    logger.go:42: 09:48:37 | watcher-tls-certs-change/0-cleanup-watcher
2026-03-14 09:56:26.591876 | controller |       | starting test step 0-cleanup-watcher\n    logger.go:42: 09:48:37 | watcher-tls-certs-change/0-cleanup-watcher
2026-03-14 09:56:26.591880 | controller |       | test step completed 0-cleanup-watcher\n    logger.go:42: 09:48:37 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.591885 | controller |       | starting test step 1-deploy-with-tlse\n    logger.go:42: 09:48:37 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.591889 | controller |       | Certificate:watcher-kuttl-default/watcher-internal-svc created\n    logger.go:42:
2026-03-14 09:56:26.591893 | controller |       09:48:37 | watcher-tls-certs-change/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-public-svc
2026-03-14 09:56:26.591898 | controller |       created\n    logger.go:42: 09:48:37 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.591902 | controller |       | Watcher:watcher-kuttl-default/watcher-kuttl created\n    logger.go:42: 09:48:37
2026-03-14 09:56:26.591906 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo
2026-03-14 09:56:26.591912 | controller |       pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.591916 | controller |       cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc get -n $NAMESPACE
2026-03-14 09:56:26.591921 | controller |       secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n
2026-03-14 09:56:26.591925 | controller |       \       # ensure that the svc secret and cert secret match\n        if [ \"${public_svc_cert}\"
2026-03-14 09:56:26.591929 | controller |       != \"${public_secret_cert}\" ]; then\n          exit 1\n        fi\n        \n        internal_svc_cert=$(oc
2026-03-14 09:56:26.591933 | controller |       rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n
2026-03-14 09:56:26.591943 | controller |       \       internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.591948 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:56:26.591952 | controller |       secret and cert secret match\n        if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\"
2026-03-14 09:56:26.591956 | controller |       ]; then\n          exit 1\n        fi\n        ]\n    logger.go:42: 09:48:37 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.591961 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:56:26.591965 | controller |       \   logger.go:42: 09:48:37 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:56:26.591969 | controller |       from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n    logger.go:42:
2026-03-14 09:56:26.591974 | controller |       09:48:37 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:56:26.591978 | controller |       09:48:38 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:56:26.591982 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.591986 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:56:26.591991 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.591995 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.591999 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592003 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592008 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:56:26.592012 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592016 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592022 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592027 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:48:38 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.592031 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:56:26.592035 | controller |       \   logger.go:42: 09:48:38 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:56:26.592039 | controller |       from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n    logger.go:42:
2026-03-14 09:56:26.592044 | controller |       09:48:38 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:56:26.592048 | controller |       09:48:39 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:56:26.592054 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592058 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:56:26.592062 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592067 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592071 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592075 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592079 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:56:26.592084 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592088 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592092 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592097 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:48:39 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.592101 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:56:26.592105 | controller |       \   logger.go:42: 09:48:40 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:56:26.592109 | controller |       from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n    logger.go:42:
2026-03-14 09:56:26.592114 | controller |       09:48:40 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:56:26.592118 | controller |       09:48:41 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:56:26.592122 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592126 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:56:26.592136 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592140 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592145 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592149 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592153 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:56:26.592158 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592162 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592166 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592170 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:48:41 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.592176 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:56:26.592181 | controller |       \   logger.go:42: 09:48:41 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:56:26.592185 | controller |       from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n    logger.go:42:
2026-03-14 09:56:26.592189 | controller |       09:48:41 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:56:26.592194 | controller |       09:48:42 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:56:26.592198 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592202 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:56:26.592206 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592211 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592215 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592219 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592224 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:56:26.592228 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592232 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592236 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592241 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:48:42 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.592245 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:56:26.592249 | controller |       \   logger.go:42: 09:48:42 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:56:26.592255 | controller |       from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n    logger.go:42:
2026-03-14 09:56:26.592259 | controller |       09:48:42 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:56:26.592264 | controller |       09:48:43 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:56:26.592268 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592272 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:56:26.592276 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592281 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592285 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592289 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592293 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:56:26.592298 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592302 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592306 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592310 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:48:43 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.592315 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:56:26.592321 | controller |       \   logger.go:42: 09:48:43 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:56:26.592330 | controller |       from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n    logger.go:42:
2026-03-14 09:56:26.592335 | controller |       09:48:43 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:56:26.592339 | controller |       09:48:44 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:56:26.592343 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592348 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:56:26.592352 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592356 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592360 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592365 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592369 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:56:26.592373 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592378 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592382 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592386 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:48:44 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.592390 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:56:26.592395 | controller |       \   logger.go:42: 09:48:45 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:56:26.592401 | controller |       from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n    logger.go:42:
2026-03-14 09:56:26.592405 | controller |       09:48:45 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:56:26.592409 | controller |       09:48:46 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:56:26.592416 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592420 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:56:26.592424 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592429 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592433 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592437 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592441 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:56:26.592446 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592450 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592456 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592460 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:48:46 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.592465 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:56:26.592469 | controller |       \   logger.go:42: 09:48:46 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:56:26.592475 | controller |       from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n    logger.go:42:
2026-03-14 09:56:26.592479 | controller |       09:48:46 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:56:26.592483 | controller |       09:48:47 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:56:26.592488 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592492 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:56:26.592496 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592500 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592505 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592509 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592513 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:56:26.592517 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592522 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592531 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592536 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:48:47 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.592540 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:56:26.592545 | controller |       \   logger.go:42: 09:48:47 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:56:26.592549 | controller |       from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n    logger.go:42:
2026-03-14 09:56:26.592553 | controller |       09:48:47 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:56:26.592557 | controller |       09:48:48 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:56:26.592562 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592566 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:56:26.592570 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592574 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592579 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592583 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592587 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:56:26.592591 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592609 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592614 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592618 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:48:48 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.592622 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:56:26.592627 | controller |       \   logger.go:42: 09:48:48 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:56:26.592631 | controller |       from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n    logger.go:42:
2026-03-14 09:56:26.592635 | controller |       09:48:48 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:56:26.592641 | controller |       09:48:50 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:56:26.592646 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592650 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:56:26.592654 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592659 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592663 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592669 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592673 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:56:26.592677 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592682 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592686 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592690 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:48:50 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.592695 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:56:26.592699 | controller |       \   logger.go:42: 09:48:50 | watcher-tls-certs-change/1-deploy-with-tlse | error:
2026-03-14 09:56:26.592703 | controller |       unable to upgrade connection: container not found (\"watcher-api\")\n    logger.go:42:
2026-03-14 09:56:26.592707 | controller |       09:48:50 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:56:26.592712 | controller |       09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:56:26.592716 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592720 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:56:26.592724 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592729 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592733 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592743 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:56:26.592748 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:56:26.592752 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.592756 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.592761 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:56:26.592765 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.592769 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:56:26.592774 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN
2026-03-14 09:56:26.592778 | controller |       CERTIFICATE-----\n    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.592782 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.592786 | controller |       09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.592793 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.592797 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.592802 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.592806 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.592810 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.592814 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.592819 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.592825 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.592829 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.592833 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.592837 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.592842 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.592846 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.592850 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.592855 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.592859 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.592863 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.592867 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.592872 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.592876 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.592880 | controller |       | ++ base64 --decode\n    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.592889 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:56:26.592894 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN
2026-03-14 09:56:26.592898 | controller |       CERTIFICATE-----\n    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.592903 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.592909 | controller |       09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.592913 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.592918 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.592922 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.592926 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.592931 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.592939 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.592943 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.592947 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.592951 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.592956 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.592960 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.592964 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.592968 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.592973 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.592977 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.592981 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.592985 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.592990 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.592994 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.592998 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.593003 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.593007 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.593011 | controller |       09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.593017 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.593021 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.593031 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.593036 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.593040 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.593044 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.593048 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.593053 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.593057 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.593064 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.593068 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.593072 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.593076 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.593081 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.593085 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.593089 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.593093 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.593098 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.593102 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.593106 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:48:51
2026-03-14 09:56:26.593111 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:56:26.593115 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.593119 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.593124 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.593130 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.593134 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.593139 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.593143 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.593147 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.593151 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.593156 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.593165 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.593170 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.593174 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.593180 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.593185 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.593189 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.593193 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.593198 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.593202 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.593206 | controller |       \   logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.593211 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.593215 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:56:26.593219 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN
2026-03-14 09:56:26.593225 | controller |       CERTIFICATE-----\n    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.593230 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.593234 | controller |       09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.593238 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.593242 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.593248 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.593253 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.593257 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.593261 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.593266 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.593270 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.593274 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.593279 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.593283 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.593289 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.593293 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.593298 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.593307 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.593312 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.593318 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.593322 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.593327 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.593331 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.593335 | controller |       | ++ base64 --decode\n    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.593340 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:56:26.593344 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN
2026-03-14 09:56:26.593348 | controller |       CERTIFICATE-----\n    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.593353 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.593357 | controller |       09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.593361 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.593368 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.593372 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.593376 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.593381 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.593385 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.593389 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.593393 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.593398 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.593402 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.593406 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.593411 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.593415 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.593419 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.593423 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.593428 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.593432 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.593436 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.593441 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.593450 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.593455 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.593459 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.593464 | controller |       09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.593468 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.593472 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.593477 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.593485 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.593489 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.593494 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.593498 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.593502 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.593506 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.593511 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.593516 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.593522 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.593528 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.593534 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.593539 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.593543 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.593549 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.593555 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.593561 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.593566 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:48:52
2026-03-14 09:56:26.593570 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n
2026-03-14 09:56:26.593574 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.593579 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.593583 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.593587 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.593609 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.593616 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.593623 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.593630 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.593634 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.593638 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.593643 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.593647 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.593651 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.593655 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.593660 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.593664 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.593668 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.593672 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.593677 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.593681 | controller |       \   logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.593685 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.593690 | controller |       | running command: [sh -c set -euxo pipefail\n        public_svc_cert=$(oc rsh -n
2026-03-14 09:56:26.593696 | controller |       $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n
2026-03-14 09:56:26.593700 | controller |       \       public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.593704 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:56:26.593709 | controller |       secret and cert secret match\n        if [ \"${public_svc_cert}\" != \"${public_secret_cert}\"
2026-03-14 09:56:26.593713 | controller |       ]; then\n          exit 1\n        fi\n        \n        internal_svc_cert=$(oc
2026-03-14 09:56:26.593717 | controller |       rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n
2026-03-14 09:56:26.593721 | controller |       \       internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.593726 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:56:26.593730 | controller |       secret and cert secret match\n        if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\"
2026-03-14 09:56:26.593734 | controller |       ]; then\n          exit 1\n        fi\n        ]\n    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.593739 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:56:26.593747 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN
2026-03-14 09:56:26.593751 | controller |       CERTIFICATE-----\n    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.593755 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.593760 | controller |       09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.593764 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.593774 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.593778 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.593783 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.593787 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.593791 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.593796 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.593800 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.593804 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.593808 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.593815 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.593819 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.593823 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.593828 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.593832 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.593836 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.593840 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.593846 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.593851 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.593855 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.593859 | controller |       | ++ base64 --decode\n    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.593865 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:56:26.593870 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN
2026-03-14 09:56:26.593874 | controller |       CERTIFICATE-----\n    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.593878 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.593882 | controller |       09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.593887 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.593891 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.593895 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.593899 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.593904 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.593913 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.593918 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.593922 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.593927 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.593931 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.593935 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.593940 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.593944 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.593948 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.593955 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.593959 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.593963 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.593967 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.593972 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.593976 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.593982 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.593987 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.593991 | controller |       09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.593995 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.593999 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.594004 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.594008 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.594012 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.594016 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.594021 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.594025 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.594029 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.594033 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.594038 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.594047 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.594051 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.594056 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.594060 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.594064 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.594069 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.594073 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.594077 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.594082 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:48:53
2026-03-14 09:56:26.594086 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:56:26.594090 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.594098 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.594103 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.594107 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.594111 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.594116 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.594120 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.594124 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.594128 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.594133 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.594137 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.594141 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.594145 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.594150 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.594154 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.594158 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.594163 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.594167 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.594171 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.594175 | controller |       \   logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.594185 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.594190 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:56:26.594194 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN
2026-03-14 09:56:26.594198 | controller |       CERTIFICATE-----\n    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.594203 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.594209 | controller |       09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.594213 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.594217 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.594221 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.594226 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.594232 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.594236 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.594240 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.594245 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.594249 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.594253 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.594258 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.594262 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.594266 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.594270 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.594275 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.594279 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.594283 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.594287 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.594292 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.594296 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.594300 | controller |       | ++ base64 --decode\n    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.594305 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:56:26.594309 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN
2026-03-14 09:56:26.594313 | controller |       CERTIFICATE-----\n    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.594319 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.594329 | controller |       09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.594333 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.594337 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.594342 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.594346 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.594350 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.594355 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.594359 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.594363 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.594370 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.594374 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.594378 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.594383 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.594387 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.594391 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.594396 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.594400 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.594404 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.594408 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.594413 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.594417 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.594421 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.594426 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.594430 | controller |       09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.594436 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.594440 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.594445 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.594449 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.594453 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.594463 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.594468 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.594472 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.594476 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.594481 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.594485 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.594489 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.594499 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.594504 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.594508 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.594512 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.594517 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.594521 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.594525 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.594529 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:48:54
2026-03-14 09:56:26.594534 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n
2026-03-14 09:56:26.594538 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.594542 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.594547 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.594553 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.594557 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.594561 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.594565 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.594570 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.594574 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.594578 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.594582 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.594587 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.594591 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.594614 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.594619 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.594623 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.594631 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.594636 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.594640 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.594644 | controller |       \   logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.594649 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.594653 | controller |       | running command: [sh -c set -euxo pipefail\n        public_svc_cert=$(oc rsh -n
2026-03-14 09:56:26.594657 | controller |       $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n
2026-03-14 09:56:26.594662 | controller |       \       public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.594666 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:56:26.594670 | controller |       secret and cert secret match\n        if [ \"${public_svc_cert}\" != \"${public_secret_cert}\"
2026-03-14 09:56:26.594675 | controller |       ]; then\n          exit 1\n        fi\n        \n        internal_svc_cert=$(oc
2026-03-14 09:56:26.594679 | controller |       rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n
2026-03-14 09:56:26.594683 | controller |       \       internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.594689 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:56:26.594694 | controller |       secret and cert secret match\n        if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\"
2026-03-14 09:56:26.594698 | controller |       ]; then\n          exit 1\n        fi\n        ]\n    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.594702 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:56:26.594708 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN
2026-03-14 09:56:26.594712 | controller |       CERTIFICATE-----\n    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.594717 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.594721 | controller |       09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.594725 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.594730 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.594734 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.594738 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.594742 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.594747 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.594751 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.594755 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.594759 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.594764 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.594774 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.594778 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.594783 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.594787 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.594791 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.594796 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.594800 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.594806 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.594811 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.594815 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.594819 | controller |       | ++ base64 --decode\n    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.594824 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:56:26.594828 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN
2026-03-14 09:56:26.594832 | controller |       CERTIFICATE-----\n    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.594838 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.594842 | controller |       09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.594847 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.594851 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.594857 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.594861 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.594865 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.594870 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.594874 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.594878 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.594883 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.594887 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.594891 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.594895 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.594900 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.594904 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.594913 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.594918 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.594922 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.594941 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.594946 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.594951 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.594955 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.594959 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.594965 | controller |       09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.594970 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.594974 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.594978 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.594982 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.594987 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.594991 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.594995 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.594999 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.595004 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.595008 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.595012 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.595016 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.595021 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.595025 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.595029 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.595033 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.595038 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.595042 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.595048 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.595054 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:48:55
2026-03-14 09:56:26.595059 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:56:26.595069 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.595074 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.595078 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.595082 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.595087 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.595091 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.595095 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.595100 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.595104 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.595108 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.595115 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.595119 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.595123 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.595127 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.595132 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.595136 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.595140 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.595145 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.595149 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.595153 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.595158 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.595162 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:56:26.595168 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN
2026-03-14 09:56:26.595172 | controller |       CERTIFICATE-----\n    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.595176 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.595181 | controller |       09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.595185 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.595189 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.595194 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.595198 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.595207 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.595212 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.595216 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.595221 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.595225 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.595229 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.595233 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.595238 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.595242 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.595248 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.595252 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.595257 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.595261 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.595265 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.595270 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.595274 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.595278 | controller |       | ++ base64 --decode\n    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.595286 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:56:26.595290 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN
2026-03-14 09:56:26.595294 | controller |       CERTIFICATE-----\n    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.595299 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.595303 | controller |       09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.595307 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.595312 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.595316 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.595320 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.595324 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.595329 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.595333 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.595337 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.595347 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.595351 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.595356 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.595360 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.595364 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.595368 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.595373 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.595377 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.595381 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.595389 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.595393 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.595398 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.595404 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.595408 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.595413 | controller |       09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.595417 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.595421 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.595426 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.595430 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.595434 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.595438 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.595443 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.595447 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.595451 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.595455 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.595460 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.595464 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.595468 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.595472 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.595482 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.595486 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.595491 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.595495 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.595499 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.595504 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:48:55
2026-03-14 09:56:26.595508 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n
2026-03-14 09:56:26.595514 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.595518 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.595523 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.595527 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.595533 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.595539 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.595543 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.595547 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.595551 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.595556 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.595560 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.595564 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.595568 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.595573 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.595577 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.595581 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.595586 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.595590 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.595604 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.595611 | controller |       \   logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.595616 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.595620 | controller |       | running command: [sh -c set -euxo pipefail\n        public_svc_cert=$(oc rsh -n
2026-03-14 09:56:26.595624 | controller |       $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n
2026-03-14 09:56:26.595629 | controller |       \       public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.595638 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:56:26.595643 | controller |       secret and cert secret match\n        if [ \"${public_svc_cert}\" != \"${public_secret_cert}\"
2026-03-14 09:56:26.595649 | controller |       ]; then\n          exit 1\n        fi\n        \n        internal_svc_cert=$(oc
2026-03-14 09:56:26.595654 | controller |       rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n
2026-03-14 09:56:26.595658 | controller |       \       internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.595662 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:56:26.595666 | controller |       secret and cert secret match\n        if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\"
2026-03-14 09:56:26.595671 | controller |       ]; then\n          exit 1\n        fi\n        ]\n    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.595675 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:56:26.595679 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN
2026-03-14 09:56:26.595683 | controller |       CERTIFICATE-----\n    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.595688 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.595693 | controller |       09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.595699 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.595706 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.595712 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.595718 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.595724 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.595730 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.595734 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.595741 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.595745 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.595750 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.595754 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.595758 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.595762 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.595767 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.595771 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.595777 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.595781 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.595786 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.595790 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.595801 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.595806 | controller |       | ++ base64 --decode\n    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.595810 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:56:26.595814 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN
2026-03-14 09:56:26.595819 | controller |       CERTIFICATE-----\n    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.595823 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.595827 | controller |       09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.595832 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.595836 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.595840 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.595844 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.595849 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.595853 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.595857 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.595862 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.595866 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.595870 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.595874 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.595881 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.595885 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.595889 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.595896 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.595900 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.595904 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.595909 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.595913 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.595917 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.595922 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.595926 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.595930 | controller |       09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.595935 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.595944 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.595949 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.595953 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.595957 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.595962 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.595966 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.595970 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.595975 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.595979 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.595983 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.595987 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.595992 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.595996 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.596000 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.596008 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.596012 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.596017 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.596021 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.596025 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:48:57
2026-03-14 09:56:26.596030 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:56:26.596034 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.596038 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.596042 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.596047 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.596051 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.596055 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.596059 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.596064 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.596068 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.596078 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.596082 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.596086 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.596091 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.596095 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.596099 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.596104 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.596108 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.596112 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.596118 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.596122 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.596127 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.596131 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:56:26.596135 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN
2026-03-14 09:56:26.596140 | controller |       CERTIFICATE-----\n    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.596144 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.596148 | controller |       09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.596153 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.596159 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.596163 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.596167 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.596172 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.596176 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.596180 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.596185 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.596189 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.596193 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.596197 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.596202 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.596206 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.596216 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.596221 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.596225 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.596229 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.596236 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.596240 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.596244 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.596252 | controller |       | ++ base64 --decode\n    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.596256 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:56:26.596260 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN
2026-03-14 09:56:26.596265 | controller |       CERTIFICATE-----\n    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.596270 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.596275 | controller |       09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.596279 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.596283 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.596288 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.596292 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.596296 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.596302 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.596306 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.596311 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.596315 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.596319 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.596323 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.596328 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.596332 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.596336 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.596341 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.596345 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.596351 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.596361 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.596365 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.596370 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.596374 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.596378 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.596383 | controller |       09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.596387 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.596391 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.596395 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.596400 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.596404 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.596408 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.596412 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.596417 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.596421 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.596428 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.596433 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.596437 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.596441 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.596446 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.596450 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.596454 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.596458 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.596463 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.596469 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.596473 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:48:57
2026-03-14 09:56:26.596478 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n
2026-03-14 09:56:26.596482 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.596486 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.596490 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.596500 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.596504 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.596509 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.596513 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.596517 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.596522 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.596526 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.596530 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.596534 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.596539 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.596545 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.596550 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.596556 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.596562 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.596568 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.596574 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.596580 | controller |       \   logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.596585 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:48:58 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.596604 | controller |       | running command: [sh -c set -euxo pipefail\n        public_svc_cert=$(oc rsh -n
2026-03-14 09:56:26.596611 | controller |       $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n
2026-03-14 09:56:26.596616 | controller |       \       public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.596620 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:56:26.596624 | controller |       secret and cert secret match\n        if [ \"${public_svc_cert}\" != \"${public_secret_cert}\"
2026-03-14 09:56:26.596628 | controller |       ]; then\n          exit 1\n        fi\n        \n        internal_svc_cert=$(oc
2026-03-14 09:56:26.596633 | controller |       rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n
2026-03-14 09:56:26.596637 | controller |       \       internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.596641 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:56:26.596645 | controller |       secret and cert secret match\n        if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\"
2026-03-14 09:56:26.596650 | controller |       ]; then\n          exit 1\n        fi\n        ]\n    logger.go:42: 09:48:58 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.596654 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:56:26.596658 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN
2026-03-14 09:56:26.596662 | controller |       CERTIFICATE-----\n    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.596667 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.596671 | controller |       09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.596682 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.596687 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.596691 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.596695 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.596700 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.596704 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.596708 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.596713 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.596717 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.596721 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.596726 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.596730 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.596737 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.596742 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.596746 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.596750 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.596757 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.596761 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.596766 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.596770 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.596774 | controller |       | ++ base64 --decode\n    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.596780 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:56:26.596785 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN
2026-03-14 09:56:26.596789 | controller |       CERTIFICATE-----\n    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.596793 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.596798 | controller |       09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.596802 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.596806 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.596811 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.596815 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.596825 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.596830 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.596834 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.596838 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.596842 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.596847 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.596851 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.596857 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.596862 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.596866 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.596870 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.596875 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.596879 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.596883 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.596888 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.596892 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.596896 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.596900 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.596905 | controller |       09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.596909 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.596913 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.596918 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.596924 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.596928 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.596932 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.596936 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.596941 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.596945 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.596949 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.596959 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.596963 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.596968 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.596974 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.596978 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.596982 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.596987 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.596991 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.596995 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.597000 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:48:59
2026-03-14 09:56:26.597004 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:56:26.597008 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.597013 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.597017 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.597021 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.597025 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.597030 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.597034 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.597040 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.597045 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.597049 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.597053 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.597057 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.597062 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.597066 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.597070 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.597074 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.597082 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.597086 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.597095 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.597100 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.597104 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.597109 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:56:26.597113 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN
2026-03-14 09:56:26.597117 | controller |       CERTIFICATE-----\n    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.597122 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.597126 | controller |       09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.597130 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.597134 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.597139 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.597143 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.597147 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.597151 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.597156 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.597160 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.597164 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.597169 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.597175 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.597179 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.597184 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.597188 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.597194 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.597198 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.597202 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.597207 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.597211 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.597215 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.597219 | controller |       | ++ base64 --decode\n    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.597224 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:56:26.597228 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN
2026-03-14 09:56:26.597232 | controller |       CERTIFICATE-----\n    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.597242 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.597246 | controller |       09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.597251 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.597255 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.597259 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.597264 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.597268 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.597272 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.597277 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.597281 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.597285 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.597289 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.597294 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.597298 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.597302 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.597306 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.597314 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.597319 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.597323 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.597327 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.597331 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.597336 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.597340 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.597344 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.597349 | controller |       09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.597353 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.597357 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.597361 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.597366 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.597375 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.597379 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.597384 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.597388 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.597392 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.597397 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.597401 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.597405 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.597409 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.597414 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.597418 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.597424 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.597428 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.597433 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.597437 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.597441 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:48:59
2026-03-14 09:56:26.597446 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n
2026-03-14 09:56:26.597450 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.597456 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.597460 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.597465 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.597469 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.597473 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.597477 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.597482 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.597486 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.597490 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.597494 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.597499 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.597508 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.597512 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.597517 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.597521 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.597525 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.597529 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.597535 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.597540 | controller |       \   logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.597544 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:49:00 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.597549 | controller |       | running command: [sh -c set -euxo pipefail\n        public_svc_cert=$(oc rsh -n
2026-03-14 09:56:26.597553 | controller |       $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n
2026-03-14 09:56:26.597559 | controller |       \       public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.597563 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:56:26.597568 | controller |       secret and cert secret match\n        if [ \"${public_svc_cert}\" != \"${public_secret_cert}\"
2026-03-14 09:56:26.597572 | controller |       ]; then\n          exit 1\n        fi\n        \n        internal_svc_cert=$(oc
2026-03-14 09:56:26.597576 | controller |       rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n
2026-03-14 09:56:26.597580 | controller |       \       internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.597585 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:56:26.597589 | controller |       secret and cert secret match\n        if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\"
2026-03-14 09:56:26.597604 | controller |       ]; then\n          exit 1\n        fi\n        ]\n    logger.go:42: 09:49:00 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.597611 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:56:26.597616 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN
2026-03-14 09:56:26.597622 | controller |       CERTIFICATE-----\n    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.597630 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.597636 | controller |       09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.597641 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.597647 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.597651 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.597656 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.597660 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.597665 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.597669 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.597673 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.597677 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.597690 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.597695 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.597699 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.597704 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.597708 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.597712 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.597717 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.597721 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.597725 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.597729 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.597736 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.597742 | controller |       | ++ base64 --decode\n    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.597746 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:56:26.597750 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN
2026-03-14 09:56:26.597755 | controller |       CERTIFICATE-----\n    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.597759 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.597763 | controller |       09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.597768 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.597772 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.597776 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.597781 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.597785 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.597789 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.597794 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.597798 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.597806 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.597810 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.597814 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.597818 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.597823 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.597832 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.597837 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.597841 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.597845 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.597850 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.597854 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.597860 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.597866 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.597872 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.597878 | controller |       09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.597882 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.597887 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.597893 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.597899 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.597905 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.597909 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.597913 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.597918 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.597922 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.597926 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.597934 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.597939 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.597945 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.597950 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.597954 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.597958 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.597962 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.597967 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.597971 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.597975 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:49:01
2026-03-14 09:56:26.597986 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:56:26.597990 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.597995 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.597999 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.598004 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.598010 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.598015 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.598019 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.598024 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.598028 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.598032 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.598036 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.598041 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.598045 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.598052 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.598056 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.598060 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.598066 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.598071 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.598075 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.598079 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.598083 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.598088 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:56:26.598092 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN
2026-03-14 09:56:26.598096 | controller |       CERTIFICATE-----\n    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.598101 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.598105 | controller |       09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.598109 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.598113 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.598118 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.598127 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.598132 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.598136 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.598140 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.598145 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.598149 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.598153 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.598158 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.598162 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.598168 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.598173 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.598177 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.598181 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.598185 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.598190 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.598194 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.598198 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.598203 | controller |       | ++ base64 --decode\n    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.598207 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:56:26.598211 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN
2026-03-14 09:56:26.598216 | controller |       CERTIFICATE-----\n    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.598220 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.598224 | controller |       09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.598228 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.598233 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.598239 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.598244 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.598248 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.598252 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.598256 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.598266 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.598270 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.598275 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.598279 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.598286 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.598290 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.598295 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.598299 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.598303 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.598307 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.598314 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.598318 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.598322 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.598327 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.598331 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.598335 | controller |       09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.598339 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.598344 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.598348 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.598352 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.598357 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.598361 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.598367 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.598371 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.598375 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.598380 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.598384 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.598388 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.598394 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.598404 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.598408 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.598413 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.598417 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.598422 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.598426 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.598430 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:49:01
2026-03-14 09:56:26.598435 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n
2026-03-14 09:56:26.598439 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.598443 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.598447 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.598452 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.598456 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.598460 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.598465 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.598469 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.598473 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.598478 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.598484 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.598488 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.598492 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.598497 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.598501 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.598507 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.598511 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.598516 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.598520 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.598524 | controller |       \   logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.598529 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.598533 | controller |       | running command: [sh -c set -euxo pipefail\n        public_svc_cert=$(oc rsh -n
2026-03-14 09:56:26.598543 | controller |       $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n
2026-03-14 09:56:26.598548 | controller |       \       public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.598552 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:56:26.598556 | controller |       secret and cert secret match\n        if [ \"${public_svc_cert}\" != \"${public_secret_cert}\"
2026-03-14 09:56:26.598561 | controller |       ]; then\n          exit 1\n        fi\n        \n        internal_svc_cert=$(oc
2026-03-14 09:56:26.598565 | controller |       rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n
2026-03-14 09:56:26.598569 | controller |       \       internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.598574 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:56:26.598578 | controller |       secret and cert secret match\n        if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\"
2026-03-14 09:56:26.598582 | controller |       ]; then\n          exit 1\n        fi\n        ]\n    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.598587 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:56:26.598591 | controller |       \   logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN
2026-03-14 09:56:26.598608 | controller |       CERTIFICATE-----\n    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.598613 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.598617 | controller |       09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.598621 | controller |       \   logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.598626 | controller |       \   logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.598630 | controller |       \   logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.598634 | controller |       \   logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.598639 | controller |       \   logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.598643 | controller |       \   logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.598649 | controller |       \   logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.598653 | controller |       \   logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.598658 | controller |       \   logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.598662 | controller |       \   logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.598666 | controller |       \   logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.598670 | controller |       \   logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.598675 | controller |       \   logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.598681 | controller |       \   logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.598685 | controller |       \   logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.598690 | controller |       \   logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.598694 | controller |       \   logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.598704 | controller |       \   logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.598709 | controller |       \   logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.598713 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.598717 | controller |       | ++ base64 --decode\n    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.598722 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:56:26.598726 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN
2026-03-14 09:56:26.598730 | controller |       CERTIFICATE-----\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.598735 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.598739 | controller |       09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.598743 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.598747 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.598752 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.598756 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.598760 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.598765 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.598772 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.598776 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.598781 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.598785 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.598789 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.598793 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.598798 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.598802 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.598806 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.598811 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.598815 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.598821 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.598825 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.598831 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.598836 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.598840 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.598844 | controller |       09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.598854 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.598858 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.598863 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.598867 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.598871 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.598876 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.598880 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.598887 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.598891 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.598895 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.598899 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.598904 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.598908 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.598912 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.598917 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.598921 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.598925 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.598946 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.598951 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.598955 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:49:03
2026-03-14 09:56:26.598959 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:56:26.598964 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.598968 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.598972 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.598976 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.598983 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.598988 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.598992 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.598996 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.599006 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.599010 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.599017 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.599021 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.599025 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.599030 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.599034 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.599038 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.599043 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.599047 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.599051 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.599055 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.599060 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.599064 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:56:26.599068 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN
2026-03-14 09:56:26.599073 | controller |       CERTIFICATE-----\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.599077 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.599081 | controller |       09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.599085 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.599090 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.599094 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.599098 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.599103 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.599107 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.599111 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.599118 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.599122 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.599128 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.599133 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.599137 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.599147 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.599151 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.599155 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.599160 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.599164 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.599168 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.599173 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.599177 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.599181 | controller |       | ++ base64 --decode\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.599186 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:56:26.599190 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN
2026-03-14 09:56:26.599196 | controller |       CERTIFICATE-----\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.599201 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.599205 | controller |       09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.599209 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.599213 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.599218 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.599222 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.599226 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.599230 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.599235 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.599239 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.599245 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.599249 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.599254 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.599260 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.599264 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.599268 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.599272 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.599277 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.599287 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.599291 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.599295 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.599300 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.599304 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.599308 | controller |       | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:56:26.599313 | controller |       09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.599317 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.599321 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.599326 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.599330 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.599334 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.599339 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.599343 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.599347 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.599351 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.599356 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.599362 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.599367 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.599371 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.599375 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.599380 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.599386 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.599390 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.599394 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.599399 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.599403 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:49:03
2026-03-14 09:56:26.599407 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa\n
2026-03-14 09:56:26.599412 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw\n
2026-03-14 09:56:26.599416 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU\n
2026-03-14 09:56:26.599425 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn\n
2026-03-14 09:56:26.599430 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb\n
2026-03-14 09:56:26.599434 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve\n
2026-03-14 09:56:26.599439 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2\n
2026-03-14 09:56:26.599443 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f\n
2026-03-14 09:56:26.599447 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:56:26.599451 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD\n
2026-03-14 09:56:26.599456 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:56:26.599460 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t\n
2026-03-14 09:56:26.599464 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa\n
2026-03-14 09:56:26.599469 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx\n
2026-03-14 09:56:26.599475 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo\n
2026-03-14 09:56:26.599479 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj\n
2026-03-14 09:56:26.599483 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok\n
2026-03-14 09:56:26.599488 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g\n
2026-03-14 09:56:26.599492 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+\n
2026-03-14 09:56:26.599496 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy\n
2026-03-14 09:56:26.599501 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.599505 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:56:26.599509 | controller |       | test step completed 1-deploy-with-tlse\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599514 | controller |       | starting test step 2-change-public-svc-certificate\n    logger.go:42: 09:49:03
2026-03-14 09:56:26.599518 | controller |       | watcher-tls-certs-change/2-change-public-svc-certificate | running command: [sh
2026-03-14 09:56:26.599522 | controller |       -c oc patch Certificate -n $NAMESPACE watcher-public-svc --type='json' -p='[{\"op\":
2026-03-14 09:56:26.599527 | controller |       \"replace\", \"path\": \"/spec/dnsNames\", \"value\":['watcher-public.watcher-kuttl-default.svc',
2026-03-14 09:56:26.599531 | controller |       'watcher-public.watcher-kuttl-default.svc.cluster.local']}]'\n        ]\n    logger.go:42:
2026-03-14 09:56:26.599535 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | certificate.cert-manager.io/watcher-public-svc
2026-03-14 09:56:26.599540 | controller |       patched\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599544 | controller |       | running command: [sh -c set -euxo pipefail\n        svc_cert=$(oc rsh -n $NAMESPACE
2026-03-14 09:56:26.599548 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        secret_cert=$(oc
2026-03-14 09:56:26.599552 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.599557 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.599561 | controller |       \       if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n          exit 1\n        fi\n
2026-03-14 09:56:26.599565 | controller |       \       ]\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599575 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:56:26.599579 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599583 | controller |       | + svc_cert='-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599588 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.599592 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.599616 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599622 | controller |       | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n    logger.go:42:
2026-03-14 09:56:26.599626 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.599631 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599635 | controller |       | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n    logger.go:42:
2026-03-14 09:56:26.599642 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.599647 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599651 | controller |       | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n    logger.go:42:
2026-03-14 09:56:26.599655 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.599660 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599664 | controller |       | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n    logger.go:42:
2026-03-14 09:56:26.599668 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.599672 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599677 | controller |       | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n    logger.go:42:
2026-03-14 09:56:26.599681 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.599687 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599692 | controller |       | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n    logger.go:42:
2026-03-14 09:56:26.599696 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.599700 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599704 | controller |       | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n    logger.go:42:
2026-03-14 09:56:26.599709 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.599713 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599717 | controller |       | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n    logger.go:42:
2026-03-14 09:56:26.599722 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.599726 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599730 | controller |       | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n    logger.go:42:
2026-03-14 09:56:26.599734 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.599738 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599743 | controller |       | -----END CERTIFICATE-----'\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599747 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:56:26.599751 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599757 | controller |       | ++ base64 --decode\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599767 | controller |       | + secret_cert='-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599772 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.599776 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.599780 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599785 | controller |       | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n    logger.go:42:
2026-03-14 09:56:26.599789 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.599793 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599798 | controller |       | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n    logger.go:42:
2026-03-14 09:56:26.599802 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.599806 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599810 | controller |       | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n    logger.go:42:
2026-03-14 09:56:26.599815 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.599819 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599823 | controller |       | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n    logger.go:42:
2026-03-14 09:56:26.599827 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.599832 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599836 | controller |       | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n    logger.go:42:
2026-03-14 09:56:26.599840 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.599844 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599849 | controller |       | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n    logger.go:42:
2026-03-14 09:56:26.599853 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.599857 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599862 | controller |       | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n    logger.go:42:
2026-03-14 09:56:26.599866 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.599870 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599874 | controller |       | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n    logger.go:42:
2026-03-14 09:56:26.599879 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.599883 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599887 | controller |       | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n    logger.go:42:
2026-03-14 09:56:26.599893 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.599898 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599902 | controller |       | -----END CERTIFICATE-----'\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599906 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599910 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.599915 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n
2026-03-14 09:56:26.599921 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599925 | controller |       | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n    logger.go:42:
2026-03-14 09:56:26.599929 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n
2026-03-14 09:56:26.599939 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599943 | controller |       | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n    logger.go:42:
2026-03-14 09:56:26.599948 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n
2026-03-14 09:56:26.599952 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599956 | controller |       | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n    logger.go:42:
2026-03-14 09:56:26.599962 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n
2026-03-14 09:56:26.599967 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599971 | controller |       | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n    logger.go:42:
2026-03-14 09:56:26.599975 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n
2026-03-14 09:56:26.599980 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599984 | controller |       | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n    logger.go:42:
2026-03-14 09:56:26.599988 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n
2026-03-14 09:56:26.599992 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.599997 | controller |       | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n    logger.go:42:
2026-03-14 09:56:26.600001 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n
2026-03-14 09:56:26.600005 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.600009 | controller |       | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n    logger.go:42:
2026-03-14 09:56:26.600014 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n
2026-03-14 09:56:26.600018 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.600022 | controller |       | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n    logger.go:42:
2026-03-14 09:56:26.600028 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n
2026-03-14 09:56:26.600033 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.600038 | controller |       | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n    logger.go:42:
2026-03-14 09:56:26.600044 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n
2026-03-14 09:56:26.600050 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.600056 | controller |       | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42:
2026-03-14 09:56:26.600063 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:56:26.600069 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.600075 | controller |       | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz\n    logger.go:42:
2026-03-14 09:56:26.600080 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL\n
2026-03-14 09:56:26.600084 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.600088 | controller |       | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM\n    logger.go:42:
2026-03-14 09:56:26.600093 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp\n
2026-03-14 09:56:26.600097 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.600101 | controller |       | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs\n    logger.go:42:
2026-03-14 09:56:26.600105 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/\n
2026-03-14 09:56:26.600109 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.600114 | controller |       | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR\n    logger.go:42:
2026-03-14 09:56:26.600124 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:56:26.600129 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.600133 | controller |       | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX\n    logger.go:42:
2026-03-14 09:56:26.600137 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:56:26.600142 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.600148 | controller |       | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh\n    logger.go:42:
2026-03-14 09:56:26.600153 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg\n
2026-03-14 09:56:26.600157 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.600162 | controller |       | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO\n    logger.go:42:
2026-03-14 09:56:26.600166 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK\n
2026-03-14 09:56:26.600170 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.600174 | controller |       | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f\n    logger.go:42:
2026-03-14 09:56:26.600181 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC\n
2026-03-14 09:56:26.600185 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.600189 | controller |       | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T\n    logger.go:42:
2026-03-14 09:56:26.600194 | controller |       09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA\n
2026-03-14 09:56:26.600198 | controller |       \   logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.600202 | controller |       | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm\n    logger.go:42: 09:49:03
2026-03-14 09:56:26.600207 | controller |       | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----'
2026-03-14 09:56:26.600211 | controller |       ']'\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.600215 | controller |       | test step completed 2-change-public-svc-certificate\n    logger.go:42: 09:49:03
2026-03-14 09:56:26.600219 | controller |       | watcher-tls-certs-change/3-change-internal-svc-certificate | starting test step
2026-03-14 09:56:26.600224 | controller |       3-change-internal-svc-certificate\n    logger.go:42: 09:49:03 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600228 | controller |       | running command: [sh -c oc patch Certificate -n $NAMESPACE watcher-internal-svc
2026-03-14 09:56:26.600232 | controller |       --type='json' -p='[{\"op\": \"replace\", \"path\": \"/spec/dnsNames\", \"value\":['watcher-internal.watcher-kuttl-default.svc',
2026-03-14 09:56:26.600236 | controller |       'watcher-internal.watcher-kuttl-default.svc.cluster.local']}]'\n        ]\n    logger.go:42:
2026-03-14 09:56:26.600241 | controller |       09:49:04 | watcher-tls-certs-change/3-change-internal-svc-certificate | certificate.cert-manager.io/watcher-internal-svc
2026-03-14 09:56:26.600245 | controller |       patched\n    logger.go:42: 09:49:04 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600249 | controller |       | running command: [sh -c set -euxo pipefail\n        svc_cert=$(oc rsh -n $NAMESPACE
2026-03-14 09:56:26.600254 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        secret_cert=$(oc
2026-03-14 09:56:26.600258 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.600262 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.600266 | controller |       \       if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n          exit 1\n        fi\n
2026-03-14 09:56:26.600271 | controller |       \       ]\n    logger.go:42: 09:49:04 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600275 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:56:26.600279 | controller |       \   logger.go:42: 09:49:05 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600284 | controller |       | error: Internal error occurred: error executing command in container: container
2026-03-14 09:56:26.600288 | controller |       is not created or running\n    logger.go:42: 09:49:05 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600292 | controller |       | + svc_cert=\n    logger.go:42: 09:49:06 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600301 | controller |       | running command: [sh -c set -euxo pipefail\n        svc_cert=$(oc rsh -n $NAMESPACE
2026-03-14 09:56:26.600308 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        secret_cert=$(oc
2026-03-14 09:56:26.600312 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.600316 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.600323 | controller |       \       if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n          exit 1\n        fi\n
2026-03-14 09:56:26.600327 | controller |       \       ]\n    logger.go:42: 09:49:06 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600331 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:56:26.600336 | controller |       \   logger.go:42: 09:49:06 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600340 | controller |       | error: cannot exec into a container in a completed pod; current phase is Failed\n
2026-03-14 09:56:26.600344 | controller |       \   logger.go:42: 09:49:06 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600349 | controller |       | + svc_cert=\n    logger.go:42: 09:49:07 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600353 | controller |       | running command: [sh -c set -euxo pipefail\n        svc_cert=$(oc rsh -n $NAMESPACE
2026-03-14 09:56:26.600357 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        secret_cert=$(oc
2026-03-14 09:56:26.600361 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.600365 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.600370 | controller |       \       if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n          exit 1\n        fi\n
2026-03-14 09:56:26.600374 | controller |       \       ]\n    logger.go:42: 09:49:07 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600378 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:56:26.600383 | controller |       \   logger.go:42: 09:49:07 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600387 | controller |       | error: unable to upgrade connection: container not found (\"watcher-api\")\n    logger.go:42:
2026-03-14 09:56:26.600391 | controller |       09:49:07 | watcher-tls-certs-change/3-change-internal-svc-certificate | + svc_cert=\n
2026-03-14 09:56:26.600395 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600400 | controller |       | running command: [sh -c set -euxo pipefail\n        svc_cert=$(oc rsh -n $NAMESPACE
2026-03-14 09:56:26.600404 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        secret_cert=$(oc
2026-03-14 09:56:26.600410 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:56:26.600414 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:56:26.600418 | controller |       \       if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n          exit 1\n        fi\n
2026-03-14 09:56:26.600423 | controller |       \       ]\n    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600427 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:56:26.600431 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600437 | controller |       | + svc_cert='-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600441 | controller |       | MIID8TCCAlmgAwIBAgIRANEJGjnrkU5zntzPj7vLEQUwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.600446 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5NDkwNFoXDTMx\n
2026-03-14 09:56:26.600451 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600456 | controller |       | MDMxMzA5NDkwNFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK2c\n    logger.go:42:
2026-03-14 09:56:26.600460 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | pgzS28NUzYI+nLrg8YANmwvnjV799rH8Mt2MKso4tswc2tn/+L4eZmTkxibhbs2M\n
2026-03-14 09:56:26.600467 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600471 | controller |       | TwKhPdGOsTdO+5uf3eq5uYBGUJJQzLg6WUjlaIE4/y8ELdXwn9Si8JNGg8TUO2+o\n    logger.go:42:
2026-03-14 09:56:26.600476 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | dJJous/W/rG11NUGJeaOJQkEuTNvmVEq3wevESe9WJhTBdxOmtfR1V6qHbL9Bbi6\n
2026-03-14 09:56:26.600480 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600490 | controller |       | djRSEFWTYTojAtbB+9ZluXXXr986tLaYQi7hrc8VhNp9wAYxnmWAV6h6OtQsv4oR\n    logger.go:42:
2026-03-14 09:56:26.600494 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | L5nRqzNvLEnrIxSN3XM2kQQL/yN7rh1fb3j8bNUtB0opUjm0SSFsP0/k0fmE4WMu\n
2026-03-14 09:56:26.600499 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600503 | controller |       | A5Sg1np9ujvpUTwAq58CAwEAAaOByzCByDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n    logger.go:42:
2026-03-14 09:56:26.600507 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBTKGZDWkzhx\n
2026-03-14 09:56:26.600512 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600516 | controller |       | g5S2LtKCE5EQO2rWmjByBgNVHREBAf8EaDBmgip3YXRjaGVyLWludGVybmFsLndh\n    logger.go:42:
2026-03-14 09:56:26.600520 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmOCOHdhdGNoZXItaW50ZXJuYWwud2F0Y2hl\n
2026-03-14 09:56:26.600525 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600529 | controller |       | ci1rdXR0bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUA\n    logger.go:42:
2026-03-14 09:56:26.600533 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | A4IBgQDNKBRriKybBx92y2bamYy1IsYsPZ9IQSIkLavcYm+ytlMKKv1bjgJ13AWo\n
2026-03-14 09:56:26.600538 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600542 | controller |       | fts0EPcJfblCY6i6hkFYh6GXuAlrW5+HiEsGcybOtH/tYVksh3qzDUKmKgJsPvAA\n    logger.go:42:
2026-03-14 09:56:26.600546 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | lzWsTfnTzgvmQK+Cfh6dg7PwOgkoB8Lv+vnYzbX9gL9Hw6qYObv/dgoGJTuj4tiG\n
2026-03-14 09:56:26.600550 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600555 | controller |       | kn4mVPYVCdpAQ5jQW9iGEuiLYvQ6/klSOD4YF2JqF3wf759g9yBOacFSIbd4xn9I\n    logger.go:42:
2026-03-14 09:56:26.600559 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | uMwr9eYjZJX1N2Yg8nYtBpi9WupGIWfBVf+7BygXxozT1KG+D6+bWS4J2Ngx6jbo\n
2026-03-14 09:56:26.600563 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600568 | controller |       | SN+rmc+mwymN7+0GLTYxXH9vEJgO1648TGlSn5S4f+ONz8av68Qwo1RsYHLIHI9h\n    logger.go:42:
2026-03-14 09:56:26.600572 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | xhsX0pMrarGyXfryX9YLnnxxpYQ4V90j8AxPawoFpj+8gNXa6X174+dKfA2QuNUQ\n
2026-03-14 09:56:26.600576 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600581 | controller |       | h6yFKf0wvwih9nnYS8SsyxhZTDZP70pYQqdziTjXNUsctIn+WGrZcsZXoMqTF179\n    logger.go:42:
2026-03-14 09:56:26.600585 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | +SCIwhM=\n
2026-03-14 09:56:26.600589 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600605 | controller |       | -----END CERTIFICATE-----'\n    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600614 | controller |       | ++ base64 --decode\n    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600621 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:56:26.600625 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600629 | controller |       | + secret_cert='-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600634 | controller |       | MIID8TCCAlmgAwIBAgIRANEJGjnrkU5zntzPj7vLEQUwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.600638 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5NDkwNFoXDTMx\n
2026-03-14 09:56:26.600642 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600647 | controller |       | MDMxMzA5NDkwNFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK2c\n    logger.go:42:
2026-03-14 09:56:26.600651 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | pgzS28NUzYI+nLrg8YANmwvnjV799rH8Mt2MKso4tswc2tn/+L4eZmTkxibhbs2M\n
2026-03-14 09:56:26.600655 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600661 | controller |       | TwKhPdGOsTdO+5uf3eq5uYBGUJJQzLg6WUjlaIE4/y8ELdXwn9Si8JNGg8TUO2+o\n    logger.go:42:
2026-03-14 09:56:26.600665 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | dJJous/W/rG11NUGJeaOJQkEuTNvmVEq3wevESe9WJhTBdxOmtfR1V6qHbL9Bbi6\n
2026-03-14 09:56:26.600676 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600681 | controller |       | djRSEFWTYTojAtbB+9ZluXXXr986tLaYQi7hrc8VhNp9wAYxnmWAV6h6OtQsv4oR\n    logger.go:42:
2026-03-14 09:56:26.600685 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | L5nRqzNvLEnrIxSN3XM2kQQL/yN7rh1fb3j8bNUtB0opUjm0SSFsP0/k0fmE4WMu\n
2026-03-14 09:56:26.600689 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600693 | controller |       | A5Sg1np9ujvpUTwAq58CAwEAAaOByzCByDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n    logger.go:42:
2026-03-14 09:56:26.600698 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBTKGZDWkzhx\n
2026-03-14 09:56:26.600702 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600706 | controller |       | g5S2LtKCE5EQO2rWmjByBgNVHREBAf8EaDBmgip3YXRjaGVyLWludGVybmFsLndh\n    logger.go:42:
2026-03-14 09:56:26.600711 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmOCOHdhdGNoZXItaW50ZXJuYWwud2F0Y2hl\n
2026-03-14 09:56:26.600715 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600719 | controller |       | ci1rdXR0bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUA\n    logger.go:42:
2026-03-14 09:56:26.600723 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | A4IBgQDNKBRriKybBx92y2bamYy1IsYsPZ9IQSIkLavcYm+ytlMKKv1bjgJ13AWo\n
2026-03-14 09:56:26.600728 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600732 | controller |       | fts0EPcJfblCY6i6hkFYh6GXuAlrW5+HiEsGcybOtH/tYVksh3qzDUKmKgJsPvAA\n    logger.go:42:
2026-03-14 09:56:26.600736 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | lzWsTfnTzgvmQK+Cfh6dg7PwOgkoB8Lv+vnYzbX9gL9Hw6qYObv/dgoGJTuj4tiG\n
2026-03-14 09:56:26.600740 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600745 | controller |       | kn4mVPYVCdpAQ5jQW9iGEuiLYvQ6/klSOD4YF2JqF3wf759g9yBOacFSIbd4xn9I\n    logger.go:42:
2026-03-14 09:56:26.600749 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | uMwr9eYjZJX1N2Yg8nYtBpi9WupGIWfBVf+7BygXxozT1KG+D6+bWS4J2Ngx6jbo\n
2026-03-14 09:56:26.600755 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600759 | controller |       | SN+rmc+mwymN7+0GLTYxXH9vEJgO1648TGlSn5S4f+ONz8av68Qwo1RsYHLIHI9h\n    logger.go:42:
2026-03-14 09:56:26.600764 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | xhsX0pMrarGyXfryX9YLnnxxpYQ4V90j8AxPawoFpj+8gNXa6X174+dKfA2QuNUQ\n
2026-03-14 09:56:26.600768 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600772 | controller |       | h6yFKf0wvwih9nnYS8SsyxhZTDZP70pYQqdziTjXNUsctIn+WGrZcsZXoMqTF179\n    logger.go:42:
2026-03-14 09:56:26.600776 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | +SCIwhM=\n
2026-03-14 09:56:26.600780 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600785 | controller |       | -----END CERTIFICATE-----'\n    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600789 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600793 | controller |       | MIID8TCCAlmgAwIBAgIRANEJGjnrkU5zntzPj7vLEQUwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:56:26.600798 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5NDkwNFoXDTMx\n
2026-03-14 09:56:26.600802 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600806 | controller |       | MDMxMzA5NDkwNFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK2c\n    logger.go:42:
2026-03-14 09:56:26.600810 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | pgzS28NUzYI+nLrg8YANmwvnjV799rH8Mt2MKso4tswc2tn/+L4eZmTkxibhbs2M\n
2026-03-14 09:56:26.600817 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600821 | controller |       | TwKhPdGOsTdO+5uf3eq5uYBGUJJQzLg6WUjlaIE4/y8ELdXwn9Si8JNGg8TUO2+o\n    logger.go:42:
2026-03-14 09:56:26.600825 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | dJJous/W/rG11NUGJeaOJQkEuTNvmVEq3wevESe9WJhTBdxOmtfR1V6qHbL9Bbi6\n
2026-03-14 09:56:26.600830 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600834 | controller |       | djRSEFWTYTojAtbB+9ZluXXXr986tLaYQi7hrc8VhNp9wAYxnmWAV6h6OtQsv4oR\n    logger.go:42:
2026-03-14 09:56:26.600843 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | L5nRqzNvLEnrIxSN3XM2kQQL/yN7rh1fb3j8bNUtB0opUjm0SSFsP0/k0fmE4WMu\n
2026-03-14 09:56:26.600848 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600852 | controller |       | A5Sg1np9ujvpUTwAq58CAwEAAaOByzCByDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n    logger.go:42:
2026-03-14 09:56:26.600857 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBTKGZDWkzhx\n
2026-03-14 09:56:26.600861 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600865 | controller |       | g5S2LtKCE5EQO2rWmjByBgNVHREBAf8EaDBmgip3YXRjaGVyLWludGVybmFsLndh\n    logger.go:42:
2026-03-14 09:56:26.600869 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmOCOHdhdGNoZXItaW50ZXJuYWwud2F0Y2hl\n
2026-03-14 09:56:26.600874 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600879 | controller |       | ci1rdXR0bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUA\n    logger.go:42:
2026-03-14 09:56:26.600884 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | A4IBgQDNKBRriKybBx92y2bamYy1IsYsPZ9IQSIkLavcYm+ytlMKKv1bjgJ13AWo\n
2026-03-14 09:56:26.600890 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600894 | controller |       | fts0EPcJfblCY6i6hkFYh6GXuAlrW5+HiEsGcybOtH/tYVksh3qzDUKmKgJsPvAA\n    logger.go:42:
2026-03-14 09:56:26.600898 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | lzWsTfnTzgvmQK+Cfh6dg7PwOgkoB8Lv+vnYzbX9gL9Hw6qYObv/dgoGJTuj4tiG\n
2026-03-14 09:56:26.600903 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600907 | controller |       | kn4mVPYVCdpAQ5jQW9iGEuiLYvQ6/klSOD4YF2JqF3wf759g9yBOacFSIbd4xn9I\n    logger.go:42:
2026-03-14 09:56:26.600911 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | uMwr9eYjZJX1N2Yg8nYtBpi9WupGIWfBVf+7BygXxozT1KG+D6+bWS4J2Ngx6jbo\n
2026-03-14 09:56:26.600915 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600920 | controller |       | SN+rmc+mwymN7+0GLTYxXH9vEJgO1648TGlSn5S4f+ONz8av68Qwo1RsYHLIHI9h\n    logger.go:42:
2026-03-14 09:56:26.600924 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | xhsX0pMrarGyXfryX9YLnnxxpYQ4V90j8AxPawoFpj+8gNXa6X174+dKfA2QuNUQ\n
2026-03-14 09:56:26.600928 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600932 | controller |       | h6yFKf0wvwih9nnYS8SsyxhZTDZP70pYQqdziTjXNUsctIn+WGrZcsZXoMqTF179\n    logger.go:42:
2026-03-14 09:56:26.600937 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | +SCIwhM=\n
2026-03-14 09:56:26.600941 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600945 | controller |       | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42:
2026-03-14 09:56:26.600949 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | MIID8TCCAlmgAwIBAgIRANEJGjnrkU5zntzPj7vLEQUwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:56:26.600954 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600958 | controller |       | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5NDkwNFoXDTMx\n    logger.go:42:
2026-03-14 09:56:26.600962 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | MDMxMzA5NDkwNFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK2c\n
2026-03-14 09:56:26.600967 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600971 | controller |       | pgzS28NUzYI+nLrg8YANmwvnjV799rH8Mt2MKso4tswc2tn/+L4eZmTkxibhbs2M\n    logger.go:42:
2026-03-14 09:56:26.600975 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | TwKhPdGOsTdO+5uf3eq5uYBGUJJQzLg6WUjlaIE4/y8ELdXwn9Si8JNGg8TUO2+o\n
2026-03-14 09:56:26.600979 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600984 | controller |       | dJJous/W/rG11NUGJeaOJQkEuTNvmVEq3wevESe9WJhTBdxOmtfR1V6qHbL9Bbi6\n    logger.go:42:
2026-03-14 09:56:26.600988 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | djRSEFWTYTojAtbB+9ZluXXXr986tLaYQi7hrc8VhNp9wAYxnmWAV6h6OtQsv4oR\n
2026-03-14 09:56:26.600992 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.600996 | controller |       | L5nRqzNvLEnrIxSN3XM2kQQL/yN7rh1fb3j8bNUtB0opUjm0SSFsP0/k0fmE4WMu\n    logger.go:42:
2026-03-14 09:56:26.601001 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | A5Sg1np9ujvpUTwAq58CAwEAAaOByzCByDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:56:26.601005 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.601014 | controller |       | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBTKGZDWkzhx\n    logger.go:42:
2026-03-14 09:56:26.601019 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | g5S2LtKCE5EQO2rWmjByBgNVHREBAf8EaDBmgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:56:26.601027 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.601033 | controller |       | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmOCOHdhdGNoZXItaW50ZXJuYWwud2F0Y2hl\n    logger.go:42:
2026-03-14 09:56:26.601038 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUA\n
2026-03-14 09:56:26.601044 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.601050 | controller |       | A4IBgQDNKBRriKybBx92y2bamYy1IsYsPZ9IQSIkLavcYm+ytlMKKv1bjgJ13AWo\n    logger.go:42:
2026-03-14 09:56:26.601054 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | fts0EPcJfblCY6i6hkFYh6GXuAlrW5+HiEsGcybOtH/tYVksh3qzDUKmKgJsPvAA\n
2026-03-14 09:56:26.601058 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.601063 | controller |       | lzWsTfnTzgvmQK+Cfh6dg7PwOgkoB8Lv+vnYzbX9gL9Hw6qYObv/dgoGJTuj4tiG\n    logger.go:42:
2026-03-14 09:56:26.601067 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | kn4mVPYVCdpAQ5jQW9iGEuiLYvQ6/klSOD4YF2JqF3wf759g9yBOacFSIbd4xn9I\n
2026-03-14 09:56:26.601071 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.601123 | controller |       | uMwr9eYjZJX1N2Yg8nYtBpi9WupGIWfBVf+7BygXxozT1KG+D6+bWS4J2Ngx6jbo\n    logger.go:42:
2026-03-14 09:56:26.601133 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | SN+rmc+mwymN7+0GLTYxXH9vEJgO1648TGlSn5S4f+ONz8av68Qwo1RsYHLIHI9h\n
2026-03-14 09:56:26.601137 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.601144 | controller |       | xhsX0pMrarGyXfryX9YLnnxxpYQ4V90j8AxPawoFpj+8gNXa6X174+dKfA2QuNUQ\n    logger.go:42:
2026-03-14 09:56:26.601148 | controller |       09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate | h6yFKf0wvwih9nnYS8SsyxhZTDZP70pYQqdziTjXNUsctIn+WGrZcsZXoMqTF179\n
2026-03-14 09:56:26.601153 | controller |       \   logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.601160 | controller |       | +SCIwhM=\n    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.601166 | controller |       | -----END CERTIFICATE-----' ']'\n    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.601171 | controller |       | test step completed 3-change-internal-svc-certificate\n    logger.go:42: 09:49:08
2026-03-14 09:56:26.601176 | controller |       | watcher-tls-certs-change/4-cleanup-watcher | starting test step 4-cleanup-watcher\n
2026-03-14 09:56:26.601181 | controller |       \   logger.go:42: 09:49:19 | watcher-tls-certs-change/4-cleanup-watcher | test step
2026-03-14 09:56:26.601186 | controller |       completed 4-cleanup-watcher\n    logger.go:42: 09:49:19 | watcher-tls-certs-change/5-clenaup-certs
2026-03-14 09:56:26.601195 | controller |       | starting test step 5-clenaup-certs\n    logger.go:42: 09:49:19 | watcher-tls-certs-change/5-clenaup-certs
2026-03-14 09:56:26.601201 | controller |       | test step completed 5-clenaup-certs\n    logger.go:42: 09:49:19 | watcher-tls-certs-change
2026-03-14 09:56:26.601205 | controller |       | skipping kubernetes event logging\n=== CONT  kuttl/harness/watcher-tls\n    logger.go:42:
2026-03-14 09:56:26.601209 | controller |       09:49:19 | watcher-tls | Skipping creation of user-supplied namespace: watcher-kuttl-default\n
2026-03-14 09:56:26.601214 | controller |       \   logger.go:42: 09:49:19 | watcher-tls/0-cleanup-watcher | starting test step
2026-03-14 09:56:26.601218 | controller |       0-cleanup-watcher\n    logger.go:42: 09:49:19 | watcher-tls/0-cleanup-watcher |
2026-03-14 09:56:26.601222 | controller |       test step completed 0-cleanup-watcher\n    logger.go:42: 09:49:19 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601226 | controller |       | starting test step 1-deploy-with-tlse\n    logger.go:42: 09:49:19 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601233 | controller |       | Certificate:watcher-kuttl-default/watcher-internal-svc created\n    logger.go:42:
2026-03-14 09:56:26.601237 | controller |       09:49:19 | watcher-tls/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-public-svc
2026-03-14 09:56:26.601242 | controller |       created\n    logger.go:42: 09:49:19 | watcher-tls/1-deploy-with-tlse | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:56:26.601246 | controller |       created\n    logger.go:42: 09:49:19 | watcher-tls/1-deploy-with-tlse | running command:
2026-03-14 09:56:26.601250 | controller |       [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient -- openstack
2026-03-14 09:56:26.601254 | controller |       service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc
2026-03-14 09:56:26.601259 | controller |       exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.601272 | controller |       -c Type -c ID | grep watcher| awk '{print $1}')\n        [ $(oc get -n ${NAMESPACE}
2026-03-14 09:56:26.601276 | controller |       keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [
2026-03-14 09:56:26.601281 | controller |       -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\"
2026-03-14 09:56:26.601285 | controller |       ]\n        [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o
2026-03-14 09:56:26.601289 | controller |       jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n        [ \"$(oc
2026-03-14 09:56:26.601293 | controller |       get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64
2026-03-14 09:56:26.601298 | controller |       -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2
2026-03-14 09:56:26.601302 | controller |       ]\n        # check that both endpoints have https set\n        oc exec -n ${NAMESPACE}
2026-03-14 09:56:26.601306 | controller |       openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https)
2026-03-14 09:56:26.601310 | controller |       == 2 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:56:26.601315 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:56:26.601319 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:56:26.601323 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:56:26.601327 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:56:26.601332 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:56:26.601336 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:56:26.601340 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:56:26.601344 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:56:26.601349 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:49:19 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601353 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.601357 | controller |       value -c Name -c Type\n    logger.go:42: 09:49:19 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601362 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:49:21 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601366 | controller |       | + '[' 0 == 1 ']'\n    logger.go:42: 09:49:23 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601370 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:56:26.601374 | controller |       -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n
2026-03-14 09:56:26.601379 | controller |       \       SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.601383 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n        [
2026-03-14 09:56:26.601389 | controller |       $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.601393 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl
2026-03-14 09:56:26.601398 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        [ \"$(oc get -n ${NAMESPACE} secret
2026-03-14 09:56:26.601402 | controller |       watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\"
2026-03-14 09:56:26.601406 | controller |       == 1 ]\n        [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.601412 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\"
2026-03-14 09:56:26.601416 | controller |       == 2 ]\n        # check that both endpoints have https set\n        oc exec -n ${NAMESPACE}
2026-03-14 09:56:26.601421 | controller |       openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https)
2026-03-14 09:56:26.601425 | controller |       == 2 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:56:26.601429 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:56:26.601434 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:56:26.601438 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:56:26.601442 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:56:26.601446 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:56:26.601451 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:56:26.601455 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:56:26.601459 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:56:26.601469 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:49:23 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601473 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.601478 | controller |       value -c Name -c Type\n    logger.go:42: 09:49:23 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601482 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:49:25 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601486 | controller |       | + '[' 0 == 1 ']'\n    logger.go:42: 09:49:26 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601490 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:56:26.601495 | controller |       -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n
2026-03-14 09:56:26.601499 | controller |       \       SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.601503 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n        [
2026-03-14 09:56:26.601508 | controller |       $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.601512 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl
2026-03-14 09:56:26.601516 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        [ \"$(oc get -n ${NAMESPACE} secret
2026-03-14 09:56:26.601520 | controller |       watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\"
2026-03-14 09:56:26.601525 | controller |       == 1 ]\n        [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.601529 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\"
2026-03-14 09:56:26.601533 | controller |       == 2 ]\n        # check that both endpoints have https set\n        oc exec -n ${NAMESPACE}
2026-03-14 09:56:26.601540 | controller |       openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https)
2026-03-14 09:56:26.601544 | controller |       == 2 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:56:26.601548 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:56:26.601552 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:56:26.601557 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:56:26.601561 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:56:26.601565 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:56:26.601569 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:56:26.601574 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:56:26.601578 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:56:26.601584 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:49:26 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601588 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.601607 | controller |       value -c Name -c Type\n    logger.go:42: 09:49:26 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601624 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:49:28 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601629 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:49:28 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601633 | controller |       | ++ grep watcher\n    logger.go:42: 09:49:28 | watcher-tls/1-deploy-with-tlse |
2026-03-14 09:56:26.601638 | controller |       ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.601642 | controller |       value -c Name -c Type -c ID\n    logger.go:42: 09:49:28 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601646 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:49:30 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601650 | controller |       | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa\n    logger.go:42: 09:49:30 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601655 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.601659 | controller |       \   logger.go:42: 09:49:30 | watcher-tls/1-deploy-with-tlse | + '[' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.601663 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa ']'\n    logger.go:42: 09:49:30 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601667 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:56:26.601672 | controller |       \   logger.go:42: 09:49:30 | watcher-tls/1-deploy-with-tlse | + '[' -n '' ']'\n
2026-03-14 09:56:26.601676 | controller |       \   logger.go:42: 09:49:31 | watcher-tls/1-deploy-with-tlse | running command: [sh
2026-03-14 09:56:26.601680 | controller |       -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient -- openstack
2026-03-14 09:56:26.601691 | controller |       service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc
2026-03-14 09:56:26.601696 | controller |       exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.601700 | controller |       -c Type -c ID | grep watcher| awk '{print $1}')\n        [ $(oc get -n ${NAMESPACE}
2026-03-14 09:56:26.601704 | controller |       keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [
2026-03-14 09:56:26.601708 | controller |       -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\"
2026-03-14 09:56:26.601712 | controller |       ]\n        [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o
2026-03-14 09:56:26.601720 | controller |       jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n        [ \"$(oc
2026-03-14 09:56:26.601724 | controller |       get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64
2026-03-14 09:56:26.601728 | controller |       -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2
2026-03-14 09:56:26.601733 | controller |       ]\n        # check that both endpoints have https set\n        oc exec -n ${NAMESPACE}
2026-03-14 09:56:26.601737 | controller |       openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https)
2026-03-14 09:56:26.601741 | controller |       == 2 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:56:26.601745 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:56:26.601750 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:56:26.601754 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:56:26.601758 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:56:26.601762 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:56:26.601767 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:56:26.601771 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:56:26.601775 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:56:26.601779 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:49:31 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601784 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.601788 | controller |       value -c Name -c Type\n    logger.go:42: 09:49:31 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601792 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:49:33 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601796 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:49:33 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601801 | controller |       | ++ grep watcher\n    logger.go:42: 09:49:33 | watcher-tls/1-deploy-with-tlse |
2026-03-14 09:56:26.601805 | controller |       ++ awk '{print $1}'\n    logger.go:42: 09:49:33 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601815 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.601819 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601824 | controller |       | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa\n    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601828 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.601832 | controller |       \   logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | + '[' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.601836 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa ']'\n    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601841 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:56:26.601845 | controller |       \   logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.601849 | controller |       ']'\n    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n
2026-03-14 09:56:26.601854 | controller |       \   logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1\n
2026-03-14 09:56:26.601858 | controller |       \   logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.601864 | controller |       secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n    logger.go:42:
2026-03-14 09:56:26.601868 | controller |       09:49:36 | watcher-tls/1-deploy-with-tlse | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:56:26.601872 | controller |       09:49:36 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret
2026-03-14 09:56:26.601877 | controller |       watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42:
2026-03-14 09:56:26.601881 | controller |       09:49:36 | watcher-tls/1-deploy-with-tlse | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n
2026-03-14 09:56:26.601890 | controller |       \   logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n    logger.go:42:
2026-03-14 09:56:26.601895 | controller |       09:49:36 | watcher-tls/1-deploy-with-tlse | + '[' 2 == 2 ']'\n    logger.go:42:
2026-03-14 09:56:26.601899 | controller |       09:49:36 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.601903 | controller |       -- openstack endpoint list\n    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601908 | controller |       | + grep infra-optim\n    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601912 | controller |       | ++ grep -c https\n    logger.go:42: 09:49:38 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601916 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:49:39 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.601920 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:56:26.601925 | controller |       -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n
2026-03-14 09:56:26.601929 | controller |       \       SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.601933 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n        [
2026-03-14 09:56:26.601938 | controller |       $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.601942 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl
2026-03-14 09:56:26.601946 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        [ \"$(oc get -n ${NAMESPACE} secret
2026-03-14 09:56:26.601950 | controller |       watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\"
2026-03-14 09:56:26.601955 | controller |       == 1 ]\n        [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.601959 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\"
2026-03-14 09:56:26.601963 | controller |       == 2 ]\n        # check that both endpoints have https set\n        oc exec -n ${NAMESPACE}
2026-03-14 09:56:26.601967 | controller |       openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https)
2026-03-14 09:56:26.601972 | controller |       == 2 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:56:26.601976 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:56:26.601980 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:56:26.601984 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:56:26.601989 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:56:26.601993 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:56:26.601997 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:56:26.602001 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:56:26.602006 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:56:26.602010 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:49:39 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602016 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.602020 | controller |       value -c Name -c Type\n    logger.go:42: 09:49:39 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602024 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:49:42 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602029 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:49:42 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602033 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.602037 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:49:42 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602043 | controller |       | ++ grep watcher\n    logger.go:42: 09:49:42 | watcher-tls/1-deploy-with-tlse |
2026-03-14 09:56:26.602047 | controller |       ++ awk '{print $1}'\n    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602052 | controller |       | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa\n    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602056 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.602062 | controller |       \   logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | + '[' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.602066 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa ']'\n    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602070 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:56:26.602080 | controller |       \   logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.602084 | controller |       ']'\n    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n
2026-03-14 09:56:26.602088 | controller |       \   logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.602093 | controller |       secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n    logger.go:42:
2026-03-14 09:56:26.602097 | controller |       09:49:44 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1\n    logger.go:42:
2026-03-14 09:56:26.602101 | controller |       09:49:44 | watcher-tls/1-deploy-with-tlse | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:56:26.602105 | controller |       09:49:44 | watcher-tls/1-deploy-with-tlse | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n
2026-03-14 09:56:26.602110 | controller |       \   logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n    logger.go:42:
2026-03-14 09:56:26.602114 | controller |       09:49:44 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret
2026-03-14 09:56:26.602118 | controller |       watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42:
2026-03-14 09:56:26.602122 | controller |       09:49:44 | watcher-tls/1-deploy-with-tlse | + '[' 2 == 2 ']'\n    logger.go:42:
2026-03-14 09:56:26.602127 | controller |       09:49:44 | watcher-tls/1-deploy-with-tlse | + grep infra-optim\n    logger.go:42:
2026-03-14 09:56:26.602131 | controller |       09:49:44 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.602135 | controller |       -- openstack endpoint list\n    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602139 | controller |       | ++ grep -c https\n    logger.go:42: 09:49:47 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602144 | controller |       | + '[' 2 == 2 ']'\n    logger.go:42: 09:49:47 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602148 | controller |       | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n
2026-03-14 09:56:26.602152 | controller |       \   logger.go:42: 09:49:47 | watcher-tls/1-deploy-with-tlse | + '[' '' == '' ']'\n
2026-03-14 09:56:26.602156 | controller |       \   logger.go:42: 09:49:47 | watcher-tls/1-deploy-with-tlse | + exit 0\n    logger.go:42:
2026-03-14 09:56:26.602163 | controller |       09:49:48 | watcher-tls/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:56:26.602167 | controller |       \       oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:56:26.602171 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n ${NAMESPACE}
2026-03-14 09:56:26.602175 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep
2026-03-14 09:56:26.602180 | controller |       watcher| awk '{print $1}')\n        [ $(oc get -n ${NAMESPACE} keystoneservice watcher
2026-03-14 09:56:26.602184 | controller |       -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE}
2026-03-14 09:56:26.602188 | controller |       watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n        [ \"$(oc get
2026-03-14 09:56:26.602192 | controller |       -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64
2026-03-14 09:56:26.602197 | controller |       -d|grep -c 'ssl=1')\" == 1 ]\n        [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.602201 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\"
2026-03-14 09:56:26.602205 | controller |       == 2 ]\n        # check that both endpoints have https set\n        oc exec -n ${NAMESPACE}
2026-03-14 09:56:26.602209 | controller |       openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https)
2026-03-14 09:56:26.602214 | controller |       == 2 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:56:26.602218 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:56:26.602222 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:56:26.602226 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:56:26.602231 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:56:26.602235 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:56:26.602239 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:56:26.602243 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:56:26.602249 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:56:26.602255 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:49:48 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602261 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.602267 | controller |       value -c Name -c Type\n    logger.go:42: 09:49:48 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602276 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:49:50 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602281 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:49:50 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602288 | controller |       | ++ grep watcher\n    logger.go:42: 09:49:50 | watcher-tls/1-deploy-with-tlse |
2026-03-14 09:56:26.602292 | controller |       ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.602296 | controller |       value -c Name -c Type -c ID\n    logger.go:42: 09:49:50 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602300 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:49:52 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602305 | controller |       | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa\n    logger.go:42: 09:49:52 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602309 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.602313 | controller |       \   logger.go:42: 09:49:52 | watcher-tls/1-deploy-with-tlse | + '[' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.602319 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa ']'\n    logger.go:42: 09:49:52 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602324 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:56:26.602328 | controller |       \   logger.go:42: 09:49:52 | watcher-tls/1-deploy-with-tlse | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.602333 | controller |       ']'\n    logger.go:42: 09:49:52 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1\n
2026-03-14 09:56:26.602339 | controller |       \   logger.go:42: 09:49:52 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n    logger.go:42:
2026-03-14 09:56:26.602344 | controller |       09:49:52 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret
2026-03-14 09:56:26.602350 | controller |       watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n    logger.go:42:
2026-03-14 09:56:26.602356 | controller |       09:49:53 | watcher-tls/1-deploy-with-tlse | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:56:26.602362 | controller |       09:49:53 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n    logger.go:42: 09:49:53
2026-03-14 09:56:26.602368 | controller |       | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.602374 | controller |       -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42: 09:49:53 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602378 | controller |       | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n    logger.go:42:
2026-03-14 09:56:26.602383 | controller |       09:49:53 | watcher-tls/1-deploy-with-tlse | + '[' 2 == 2 ']'\n    logger.go:42:
2026-03-14 09:56:26.602387 | controller |       09:49:53 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.602391 | controller |       -- openstack endpoint list\n    logger.go:42: 09:49:53 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602395 | controller |       | + grep infra-optim\n    logger.go:42: 09:49:53 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602400 | controller |       | ++ grep -c https\n    logger.go:42: 09:49:55 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602404 | controller |       | + '[' 2 == 2 ']'\n    logger.go:42: 09:49:55 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:56:26.602408 | controller |       | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n
2026-03-14 09:56:26.602412 | controller |       \   logger.go:42: 09:49:55 | watcher-tls/1-deploy-with-tlse | + '[' '' == '' ']'\n
2026-03-14 09:56:26.602417 | controller |       \   logger.go:42: 09:49:55 | watcher-tls/1-deploy-with-tlse | + exit 0\n    logger.go:42:
2026-03-14 09:56:26.602421 | controller |       09:49:55 | watcher-tls/1-deploy-with-tlse | test step completed 1-deploy-with-tlse\n
2026-03-14 09:56:26.602425 | controller |       \   logger.go:42: 09:49:55 | watcher-tls/2-patch-mtls | starting test step 2-patch-mtls\n
2026-03-14 09:56:26.602429 | controller |       \   logger.go:42: 09:49:55 | watcher-tls/2-patch-mtls | running command: [sh -c
2026-03-14 09:56:26.602434 | controller |       set -euxo pipefail\n        oc patch oscp -n $NAMESPACE openstack --type='json'
2026-03-14 09:56:26.602438 | controller |       -p='[{\"op\": \"replace\", \"path\": \"/spec/memcached/templates/memcached/tls/mtls/sslVerifyMode\",
2026-03-14 09:56:26.602442 | controller |       \"value\": \"Request\"}]'\n        ]\n    logger.go:42: 09:49:55 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.602449 | controller |       | + oc patch oscp -n watcher-kuttl-default openstack --type=json '-p=[{\"op\": \"replace\",
2026-03-14 09:56:26.602453 | controller |       \"path\": \"/spec/memcached/templates/memcached/tls/mtls/sslVerifyMode\", \"value\":
2026-03-14 09:56:26.602457 | controller |       \"Request\"}]'\n    logger.go:42: 09:49:55 | watcher-tls/2-patch-mtls | openstackcontrolplane.core.openstack.org/openstack
2026-03-14 09:56:26.602462 | controller |       patched\n    logger.go:42: 09:49:55 | watcher-tls/2-patch-mtls | running command:
2026-03-14 09:56:26.602466 | controller |       [sh -c set -euxo pipefail\n        \n        oc project ${NAMESPACE}\n        #
2026-03-14 09:56:26.602472 | controller |       Get pod names for each watcher service\n        APIPOD=$(oc get pods -l service=watcher-api
2026-03-14 09:56:26.602483 | controller |       -o jsonpath='{.items[0].metadata.name}')\n        APPLIERPOD=$(oc get pods -l service=watcher-applier
2026-03-14 09:56:26.602488 | controller |       -o jsonpath='{.items[0].metadata.name}')\n        DECISIONENGINEPOD=$(oc get pods
2026-03-14 09:56:26.602492 | controller |       -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n        \n
2026-03-14 09:56:26.602496 | controller |       \       # Verify memcached mTLS config parameters in watcher-api config\n        if
2026-03-14 09:56:26.602501 | controller |       [ -n \"${APIPOD}\" ]; then\n          echo \"Checking watcher-api config...\"\n
2026-03-14 09:56:26.602505 | controller |       \         [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602509 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:56:26.602513 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602517 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:56:26.602522 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602526 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:56:26.602532 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602536 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          # Verify
2026-03-14 09:56:26.602540 | controller |       mTLS config parameters in memcached backend config\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.602545 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:56:26.602549 | controller |       \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.602553 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:56:26.602557 | controller |       \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.602561 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:56:26.602566 | controller |       \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n          exit
2026-03-14 09:56:26.602570 | controller |       1\n        fi\n        \n        # Verify memcached mTLS config parameters in watcher-applier
2026-03-14 09:56:26.602574 | controller |       config\n        if [ -n \"${APPLIERPOD}\" ]; then\n          echo \"Checking watcher-applier
2026-03-14 09:56:26.602578 | controller |       config...\"\n          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602583 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:56:26.602587 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602591 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:56:26.602610 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602614 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:56:26.602618 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602623 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          [ $(echo
2026-03-14 09:56:26.602627 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602631 | controller |       | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo
2026-03-14 09:56:26.602637 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602642 | controller |       | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo
2026-03-14 09:56:26.602646 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602650 | controller |       | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n
2026-03-14 09:56:26.602655 | controller |       \         exit 1\n        fi\n        \n        # Verify memcached mTLS config parameters
2026-03-14 09:56:26.602659 | controller |       in watcher-decision-engine config\n        if [ -n \"${DECISIONENGINEPOD}\" ]; then\n
2026-03-14 09:56:26.602663 | controller |       \         echo \"Checking watcher-decision-engine config...\"\n          [ $(echo
2026-03-14 09:56:26.602667 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602672 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:56:26.602676 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602680 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:56:26.602690 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602694 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:56:26.602699 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602703 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          [ $(echo
2026-03-14 09:56:26.602707 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602712 | controller |       | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo
2026-03-14 09:56:26.602716 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602720 | controller |       | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo
2026-03-14 09:56:26.602724 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602729 | controller |       | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n
2026-03-14 09:56:26.602733 | controller |       \         exit 1\n        fi\n        ]\n    logger.go:42: 09:49:55 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.602737 | controller |       | + oc project watcher-kuttl-default\n    logger.go:42: 09:49:55 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.602741 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.602746 | controller |       \   logger.go:42: 09:49:55 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api
2026-03-14 09:56:26.602750 | controller |       -o 'jsonpath={.items[0].metadata.name}'\n    logger.go:42: 09:49:55 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.602754 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:49:55 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.602758 | controller |       | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n
2026-03-14 09:56:26.602763 | controller |       \   logger.go:42: 09:49:56 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n
2026-03-14 09:56:26.602767 | controller |       \   logger.go:42: 09:49:56 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine
2026-03-14 09:56:26.602771 | controller |       -o 'jsonpath={.items[0].metadata.name}'\n    logger.go:42: 09:49:56 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.602776 | controller |       | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n    logger.go:42: 09:49:56
2026-03-14 09:56:26.602785 | controller |       | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42:
2026-03-14 09:56:26.602789 | controller |       09:49:56 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n
2026-03-14 09:56:26.602794 | controller |       \   logger.go:42: 09:49:56 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n
2026-03-14 09:56:26.602798 | controller |       \   logger.go:42: 09:49:56 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile
2026-03-14 09:56:26.602802 | controller |       = /etc/pki/tls/certs/mtls.crt '\n    logger.go:42: 09:49:56 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.602806 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.602811 | controller |       \   logger.go:42: 09:49:58 | watcher-tls/2-patch-mtls | error: Internal error occurred:
2026-03-14 09:56:26.602815 | controller |       error executing command in container: container is not created or running\n    logger.go:42:
2026-03-14 09:56:26.602819 | controller |       09:49:58 | watcher-tls/2-patch-mtls | ++ echo\n    logger.go:42: 09:49:58 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.602823 | controller |       | + '[' 0 == 1 ']'\n    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | running
2026-03-14 09:56:26.602828 | controller |       command: [sh -c set -euxo pipefail\n        \n        oc project ${NAMESPACE}\n
2026-03-14 09:56:26.602832 | controller |       \       # Get pod names for each watcher service\n        APIPOD=$(oc get pods -l
2026-03-14 09:56:26.602836 | controller |       service=watcher-api -o jsonpath='{.items[0].metadata.name}')\n        APPLIERPOD=$(oc
2026-03-14 09:56:26.602841 | controller |       get pods -l service=watcher-applier -o jsonpath='{.items[0].metadata.name}')\n        DECISIONENGINEPOD=$(oc
2026-03-14 09:56:26.602845 | controller |       get pods -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n
2026-03-14 09:56:26.602849 | controller |       \       \n        # Verify memcached mTLS config parameters in watcher-api config\n
2026-03-14 09:56:26.602853 | controller |       \       if [ -n \"${APIPOD}\" ]; then\n          echo \"Checking watcher-api config...\"\n
2026-03-14 09:56:26.602857 | controller |       \         [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602862 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:56:26.602866 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602870 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:56:26.602880 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602884 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:56:26.602889 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602893 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          # Verify
2026-03-14 09:56:26.602897 | controller |       mTLS config parameters in memcached backend config\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.602901 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:56:26.602906 | controller |       \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.602910 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:56:26.602914 | controller |       \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.602920 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:56:26.602924 | controller |       \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n          exit
2026-03-14 09:56:26.602943 | controller |       1\n        fi\n        \n        # Verify memcached mTLS config parameters in watcher-applier
2026-03-14 09:56:26.602951 | controller |       config\n        if [ -n \"${APPLIERPOD}\" ]; then\n          echo \"Checking watcher-applier
2026-03-14 09:56:26.602957 | controller |       config...\"\n          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602962 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:56:26.602966 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602970 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:56:26.602975 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602979 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:56:26.602983 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602987 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          [ $(echo
2026-03-14 09:56:26.602992 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.602996 | controller |       | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo
2026-03-14 09:56:26.603002 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603006 | controller |       | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo
2026-03-14 09:56:26.603010 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603015 | controller |       | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n
2026-03-14 09:56:26.603019 | controller |       \         exit 1\n        fi\n        \n        # Verify memcached mTLS config parameters
2026-03-14 09:56:26.603023 | controller |       in watcher-decision-engine config\n        if [ -n \"${DECISIONENGINEPOD}\" ]; then\n
2026-03-14 09:56:26.603028 | controller |       \         echo \"Checking watcher-decision-engine config...\"\n          [ $(echo
2026-03-14 09:56:26.603032 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603036 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:56:26.603040 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603045 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:56:26.603049 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603053 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:56:26.603058 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603062 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          [ $(echo
2026-03-14 09:56:26.603068 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603072 | controller |       | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo
2026-03-14 09:56:26.603076 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603081 | controller |       | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo
2026-03-14 09:56:26.603085 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603095 | controller |       | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n
2026-03-14 09:56:26.603099 | controller |       \         exit 1\n        fi\n        ]\n    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.603105 | controller |       | + oc project watcher-kuttl-default\n    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.603110 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.603114 | controller |       \   logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api
2026-03-14 09:56:26.603118 | controller |       -o 'jsonpath={.items[0].metadata.name}'\n    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.603123 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.603127 | controller |       | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n
2026-03-14 09:56:26.603131 | controller |       \   logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n
2026-03-14 09:56:26.603136 | controller |       \   logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine
2026-03-14 09:56:26.603140 | controller |       -o 'jsonpath={.items[0].metadata.name}'\n    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.603144 | controller |       | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n    logger.go:42: 09:49:59
2026-03-14 09:56:26.603149 | controller |       | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42:
2026-03-14 09:56:26.603153 | controller |       09:49:59 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n
2026-03-14 09:56:26.603157 | controller |       \   logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n
2026-03-14 09:56:26.603161 | controller |       \   logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile
2026-03-14 09:56:26.603166 | controller |       = /etc/pki/tls/certs/mtls.crt '\n    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.603170 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.603174 | controller |       \   logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | error: unable to upgrade
2026-03-14 09:56:26.603179 | controller |       connection: container not found (\"watcher-api\")\n    logger.go:42: 09:49:59 |
2026-03-14 09:56:26.603183 | controller |       watcher-tls/2-patch-mtls | ++ echo\n    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.603187 | controller |       | + '[' 0 == 1 ']'\n    logger.go:42: 09:50:00 | watcher-tls/2-patch-mtls | running
2026-03-14 09:56:26.603192 | controller |       command: [sh -c set -euxo pipefail\n        \n        oc project ${NAMESPACE}\n
2026-03-14 09:56:26.603196 | controller |       \       # Get pod names for each watcher service\n        APIPOD=$(oc get pods -l
2026-03-14 09:56:26.603200 | controller |       service=watcher-api -o jsonpath='{.items[0].metadata.name}')\n        APPLIERPOD=$(oc
2026-03-14 09:56:26.603204 | controller |       get pods -l service=watcher-applier -o jsonpath='{.items[0].metadata.name}')\n        DECISIONENGINEPOD=$(oc
2026-03-14 09:56:26.603209 | controller |       get pods -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n
2026-03-14 09:56:26.603213 | controller |       \       \n        # Verify memcached mTLS config parameters in watcher-api config\n
2026-03-14 09:56:26.603217 | controller |       \       if [ -n \"${APIPOD}\" ]; then\n          echo \"Checking watcher-api config...\"\n
2026-03-14 09:56:26.603221 | controller |       \         [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603226 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:56:26.603230 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603234 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:56:26.603240 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603245 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:56:26.603251 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603255 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          # Verify
2026-03-14 09:56:26.603259 | controller |       mTLS config parameters in memcached backend config\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.603263 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:56:26.603268 | controller |       \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.603272 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:56:26.603276 | controller |       \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.603280 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:56:26.603290 | controller |       \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n          exit
2026-03-14 09:56:26.603294 | controller |       1\n        fi\n        \n        # Verify memcached mTLS config parameters in watcher-applier
2026-03-14 09:56:26.603299 | controller |       config\n        if [ -n \"${APPLIERPOD}\" ]; then\n          echo \"Checking watcher-applier
2026-03-14 09:56:26.603303 | controller |       config...\"\n          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603307 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:56:26.603313 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603318 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:56:26.603324 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603330 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:56:26.603336 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603342 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          [ $(echo
2026-03-14 09:56:26.603348 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603353 | controller |       | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo
2026-03-14 09:56:26.603358 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603362 | controller |       | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo
2026-03-14 09:56:26.603366 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603371 | controller |       | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n
2026-03-14 09:56:26.603375 | controller |       \         exit 1\n        fi\n        \n        # Verify memcached mTLS config parameters
2026-03-14 09:56:26.603379 | controller |       in watcher-decision-engine config\n        if [ -n \"${DECISIONENGINEPOD}\" ]; then\n
2026-03-14 09:56:26.603383 | controller |       \         echo \"Checking watcher-decision-engine config...\"\n          [ $(echo
2026-03-14 09:56:26.603388 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603392 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:56:26.603396 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603400 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:56:26.603405 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603411 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:56:26.603416 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603420 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          [ $(echo
2026-03-14 09:56:26.603426 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603431 | controller |       | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo
2026-03-14 09:56:26.603435 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603439 | controller |       | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo
2026-03-14 09:56:26.603444 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.603448 | controller |       | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n
2026-03-14 09:56:26.603452 | controller |       \         exit 1\n        fi\n        ]\n    logger.go:42: 09:50:00 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.603456 | controller |       | + oc project watcher-kuttl-default\n    logger.go:42: 09:50:00 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.603461 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.603465 | controller |       \   logger.go:42: 09:50:00 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api
2026-03-14 09:56:26.603469 | controller |       -o 'jsonpath={.items[0].metadata.name}'\n    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.603474 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.603480 | controller |       | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n
2026-03-14 09:56:26.603484 | controller |       \   logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n
2026-03-14 09:56:26.603494 | controller |       \   logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine
2026-03-14 09:56:26.603499 | controller |       -o 'jsonpath={.items[0].metadata.name}'\n    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.603503 | controller |       | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n    logger.go:42: 09:50:01
2026-03-14 09:56:26.603508 | controller |       | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42:
2026-03-14 09:56:26.603512 | controller |       09:50:01 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n
2026-03-14 09:56:26.603516 | controller |       \   logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n
2026-03-14 09:56:26.603520 | controller |       \   logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile
2026-03-14 09:56:26.603525 | controller |       = /etc/pki/tls/certs/mtls.crt '\n    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.603529 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.603533 | controller |       \   logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:56:26.603537 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.603542 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.603546 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.603551 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.603557 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.603561 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.603565 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.603569 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.603574 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.603578 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.603582 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.603587 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.603591 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.603606 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.603622 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.603627 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.603631 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.603635 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.603639 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.603644 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.603648 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.603652 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.603656 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.603661 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.603665 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.603669 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.603673 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.603678 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.603682 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.603686 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.603691 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.603701 | controller |       period = 900\n    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.603705 | controller |       1 ']'\n    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile
2026-03-14 09:56:26.603710 | controller |       = /etc/pki/tls/private/mtls.key'\n    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.603714 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.603718 | controller |       \   logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:56:26.603725 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.603729 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.603733 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.603738 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.603742 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.603746 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.603750 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.603757 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.603762 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.603766 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.603771 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.603775 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.603779 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.603783 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.603788 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.603795 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.603799 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.603804 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.603808 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.603812 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.603816 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.603821 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.603825 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.603829 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.603833 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.603838 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.603842 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.603846 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.603850 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.603855 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.603859 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.603863 | controller |       period = 900\n    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.603869 | controller |       1 ']'\n    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile
2026-03-14 09:56:26.603873 | controller |       = /etc/pki/tls/certs/mtls-ca.crt'\n    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.603878 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.603887 | controller |       \   logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:56:26.603892 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.603896 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.603901 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.603905 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.603909 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.603913 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.603918 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.603922 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.603926 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.603930 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.603935 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.603939 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.603943 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.603947 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.603951 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.603956 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.603960 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.603964 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.603968 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.603973 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.603977 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.603981 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.603985 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.603990 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.603994 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.603998 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.604003 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.604008 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.604013 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.604017 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.604021 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.604026 | controller |       period = 900\n    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.604030 | controller |       1 ']'\n    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled
2026-03-14 09:56:26.604034 | controller |       = true'\n    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.604038 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:56:26.604043 | controller |       09:50:02 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:56:26.604047 | controller |       transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.604051 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.604055 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.604067 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.604072 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.604076 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.604080 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.604084 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.604089 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.604093 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.604097 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.604101 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.604106 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.604111 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.604116 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.604120 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.604124 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.604129 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.604133 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.604137 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.604142 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.604146 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.604152 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.604156 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.604160 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.604165 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.604169 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.604173 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.604177 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.604182 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.604186 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.604192 | controller |       period = 900\n    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.604198 | controller |       1 ']'\n    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n
2026-03-14 09:56:26.604204 | controller |       \   logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.604210 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:56:26.604214 | controller |       09:50:02 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:56:26.604218 | controller |       transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.604225 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.604229 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.604233 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.604238 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.604242 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.604246 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.604257 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.604262 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.604266 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.604270 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.604275 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.604279 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.604283 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.604287 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.604292 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.604296 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.604300 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.604306 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.604311 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.604315 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.604319 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.604324 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.604328 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.604332 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.604337 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.604341 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.604347 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.604351 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.604355 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.604360 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.604364 | controller |       period = 900\n    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.604368 | controller |       1 ']'\n    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n
2026-03-14 09:56:26.604373 | controller |       \   logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.604377 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:56:26.604381 | controller |       09:50:02 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:56:26.604386 | controller |       transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.604390 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.604394 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.604399 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.604403 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.604407 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.604412 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.604416 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.604420 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.604425 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.604429 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.604433 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.604443 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.604449 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.604453 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.604458 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.604462 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.604466 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.604471 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.604475 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.604479 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.604483 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.604488 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.604492 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.604496 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.604501 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.604505 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.604509 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.604513 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.604518 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.604522 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.604526 | controller |       period = 900\n    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.604531 | controller |       1 ']'\n    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n
2026-03-14 09:56:26.604535 | controller |       \   logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.604539 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:56:26.604543 | controller |       09:50:02 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:56:26.604548 | controller |       transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.604554 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.604558 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.604562 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.604567 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.604571 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.604575 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.604579 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.604585 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.604590 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.604604 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.604611 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.604615 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.604620 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.604624 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.604630 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.604634 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.604645 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.604649 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.604654 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.604658 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.604662 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.604667 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.604671 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.604675 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.604679 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.604684 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.604688 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.604692 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.604696 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.604701 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.604705 | controller |       period = 900\n    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.604709 | controller |       1 ']'\n    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-applier-0
2026-03-14 09:56:26.604714 | controller |       ']'\n    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-applier
2026-03-14 09:56:26.604718 | controller |       config...'\n    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | Checking watcher-applier
2026-03-14 09:56:26.604722 | controller |       config...\n    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile
2026-03-14 09:56:26.604726 | controller |       = /etc/pki/tls/certs/mtls.crt '\n    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.604731 | controller |       | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.604735 | controller |       \   logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:56:26.604741 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.604747 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.604752 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.604758 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.604762 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.604767 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.604771 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.604775 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.604779 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.604784 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.604788 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.604792 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.604796 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.604801 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.604805 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.604809 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.604813 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.604818 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.604827 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.604832 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.604836 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:56:26.604840 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.604845 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.604849 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.604853 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.604857 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.604862 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.604866 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.604870 | controller |       period = 900\n    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.604875 | controller |       1 ']'\n    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile
2026-03-14 09:56:26.604879 | controller |       = /etc/pki/tls/private/mtls.key'\n    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.604883 | controller |       | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.604887 | controller |       \   logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:56:26.604895 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.604899 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.604903 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.604908 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.604912 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.604916 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.604921 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.604925 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.604931 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.604935 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.604940 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.604944 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.604950 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.604954 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.604958 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.604963 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.604967 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.604971 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.604976 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.604980 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.604984 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:56:26.604988 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.604993 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.604997 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.605001 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.605010 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.605015 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.605020 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.605024 | controller |       period = 900\n    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.605028 | controller |       1 ']'\n    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile
2026-03-14 09:56:26.605034 | controller |       = /etc/pki/tls/certs/mtls-ca.crt'\n    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.605038 | controller |       | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.605043 | controller |       \   logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:56:26.605047 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.605051 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.605056 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.605060 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.605064 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.605069 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.605073 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.605077 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.605081 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.605086 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.605090 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.605094 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.605098 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.605103 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.605107 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.605111 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.605116 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.605120 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.605124 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.605128 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.605133 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:56:26.605137 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.605141 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.605145 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.605150 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.605154 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.605158 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.605163 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.605169 | controller |       period = 900\n    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.605173 | controller |       1 ']'\n    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled
2026-03-14 09:56:26.605177 | controller |       = true'\n    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.605182 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:56:26.605193 | controller |       09:50:03 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:56:26.605198 | controller |       transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.605202 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.605207 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.605211 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.605215 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.605220 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.605224 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.605228 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.605232 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.605237 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.605241 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.605245 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.605249 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.605255 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.605261 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.605267 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.605273 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.605279 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.605285 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.605291 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.605296 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:56:26.605300 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.605304 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.605309 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.605313 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.605317 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.605324 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.605329 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.605333 | controller |       period = 900\n    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.605337 | controller |       1 ']'\n    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n
2026-03-14 09:56:26.605341 | controller |       \   logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.605353 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:56:26.605358 | controller |       09:50:04 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:56:26.605362 | controller |       transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.605371 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.605375 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.605380 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.605391 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.605395 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.605400 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.605404 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.605408 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.605413 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.605417 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.605423 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.605428 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.605432 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.605436 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.605441 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.605445 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.605449 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.605453 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.605458 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.605462 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:56:26.605466 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.605471 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.605475 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.605482 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.605486 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.605490 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.605495 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.605500 | controller |       period = 900\n    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.605505 | controller |       1 ']'\n    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n
2026-03-14 09:56:26.605509 | controller |       \   logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.605513 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:56:26.605518 | controller |       09:50:04 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:56:26.605522 | controller |       transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.605526 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.605531 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.605535 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.605539 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.605543 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.605548 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.605552 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.605556 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.605561 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.605565 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.605575 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.605579 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.605584 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.605588 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.605592 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.605611 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.605615 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.605620 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.605624 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.605628 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:56:26.605633 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.605639 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.605643 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.605647 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.605652 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.605656 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.605662 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.605666 | controller |       period = 900\n    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.605671 | controller |       1 ']'\n    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n
2026-03-14 09:56:26.605675 | controller |       \   logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.605679 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:56:26.605684 | controller |       09:50:04 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:56:26.605688 | controller |       transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.605692 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.605697 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.605701 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.605705 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.605709 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.605714 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.605718 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.605722 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.605726 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.605731 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.605735 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.605739 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.605744 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.605748 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.605752 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.605756 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.605761 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.605771 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.605775 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.605781 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:56:26.605787 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.605792 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.605796 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.605800 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.605805 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.605809 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.605813 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.605817 | controller |       period = 900\n    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.605822 | controller |       1 ']'\n    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.605826 | controller |       ']'\n    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-decision-engine
2026-03-14 09:56:26.605830 | controller |       config...'\n    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | Checking watcher-decision-engine
2026-03-14 09:56:26.605835 | controller |       config...\n    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | +++ oc rsh -c
2026-03-14 09:56:26.605839 | controller |       watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.605843 | controller |       \   logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile
2026-03-14 09:56:26.605847 | controller |       = /etc/pki/tls/certs/mtls.crt '\n    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.605852 | controller |       | error: cannot exec into a container in a completed pod; current phase is Succeeded\n
2026-03-14 09:56:26.605856 | controller |       \   logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | ++ echo\n    logger.go:42:
2026-03-14 09:56:26.605860 | controller |       09:50:04 | watcher-tls/2-patch-mtls | + '[' 0 == 1 ']'\n    logger.go:42: 09:50:05
2026-03-14 09:56:26.605864 | controller |       | watcher-tls/2-patch-mtls | running command: [sh -c set -euxo pipefail\n        \n
2026-03-14 09:56:26.605869 | controller |       \       oc project ${NAMESPACE}\n        # Get pod names for each watcher service\n
2026-03-14 09:56:26.605873 | controller |       \       APIPOD=$(oc get pods -l service=watcher-api -o jsonpath='{.items[0].metadata.name}')\n
2026-03-14 09:56:26.605879 | controller |       \       APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath='{.items[0].metadata.name}')\n
2026-03-14 09:56:26.605883 | controller |       \       DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n
2026-03-14 09:56:26.605888 | controller |       \       \n        # Verify memcached mTLS config parameters in watcher-api config\n
2026-03-14 09:56:26.605892 | controller |       \       if [ -n \"${APIPOD}\" ]; then\n          echo \"Checking watcher-api config...\"\n
2026-03-14 09:56:26.605896 | controller |       \         [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.605900 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:56:26.605905 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.605909 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:56:26.605913 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.605917 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:56:26.605930 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.605951 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          # Verify
2026-03-14 09:56:26.605960 | controller |       mTLS config parameters in memcached backend config\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.605965 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:56:26.605969 | controller |       \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.605973 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:56:26.605977 | controller |       \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.605982 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:56:26.605992 | controller |       \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n          exit
2026-03-14 09:56:26.605996 | controller |       1\n        fi\n        \n        # Verify memcached mTLS config parameters in watcher-applier
2026-03-14 09:56:26.606001 | controller |       config\n        if [ -n \"${APPLIERPOD}\" ]; then\n          echo \"Checking watcher-applier
2026-03-14 09:56:26.606005 | controller |       config...\"\n          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.606009 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:56:26.606013 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.606018 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:56:26.606022 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.606026 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:56:26.606030 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.606035 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          [ $(echo
2026-03-14 09:56:26.606039 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.606045 | controller |       | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo
2026-03-14 09:56:26.606050 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.606054 | controller |       | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo
2026-03-14 09:56:26.606059 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.606063 | controller |       | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n
2026-03-14 09:56:26.606067 | controller |       \         exit 1\n        fi\n        \n        # Verify memcached mTLS config parameters
2026-03-14 09:56:26.606072 | controller |       in watcher-decision-engine config\n        if [ -n \"${DECISIONENGINEPOD}\" ]; then\n
2026-03-14 09:56:26.606076 | controller |       \         echo \"Checking watcher-decision-engine config...\"\n          [ $(echo
2026-03-14 09:56:26.606080 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.606084 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:56:26.606089 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.606093 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:56:26.606097 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.606104 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:56:26.606108 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.606113 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          [ $(echo
2026-03-14 09:56:26.606119 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.606123 | controller |       | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo
2026-03-14 09:56:26.606127 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.606132 | controller |       | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo
2026-03-14 09:56:26.606136 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.606140 | controller |       | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n
2026-03-14 09:56:26.606145 | controller |       \         exit 1\n        fi\n        ]\n    logger.go:42: 09:50:05 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.606149 | controller |       | + oc project watcher-kuttl-default\n    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.606153 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.606158 | controller |       \   logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api
2026-03-14 09:56:26.606162 | controller |       -o 'jsonpath={.items[0].metadata.name}'\n    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.606166 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.606171 | controller |       | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n
2026-03-14 09:56:26.606175 | controller |       \   logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n
2026-03-14 09:56:26.606185 | controller |       \   logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine
2026-03-14 09:56:26.606190 | controller |       -o 'jsonpath={.items[0].metadata.name}'\n    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.606194 | controller |       | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n    logger.go:42: 09:50:06
2026-03-14 09:56:26.606198 | controller |       | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42:
2026-03-14 09:56:26.606203 | controller |       09:50:06 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n
2026-03-14 09:56:26.606207 | controller |       \   logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n
2026-03-14 09:56:26.606211 | controller |       \   logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.606216 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:56:26.606220 | controller |       09:50:06 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt
2026-03-14 09:56:26.606224 | controller |       '\n    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:56:26.606229 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.606233 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.606237 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.606242 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.606248 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.606252 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.606257 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.606261 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.606267 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.606272 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.606276 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.606280 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.606285 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.606289 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.606294 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.606298 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.606302 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.606306 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.606311 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.606317 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.606322 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.606327 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.606331 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.606336 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.606342 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.606348 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.606354 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.606360 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.606366 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.606372 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.606378 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.606400 | controller |       period = 900\n    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.606405 | controller |       1 ']'\n    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile
2026-03-14 09:56:26.606410 | controller |       = /etc/pki/tls/private/mtls.key'\n    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.606414 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.606419 | controller |       \   logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:56:26.606425 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.606429 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.606434 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.606438 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.606443 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.606447 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.606451 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.606456 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.606460 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.606464 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.606469 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.606473 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.606477 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.606482 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.606486 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.606490 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.606495 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.606499 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.606503 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.606508 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.606512 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.606516 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.606520 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.606525 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.606529 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.606534 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.606538 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.606542 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.606547 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.606551 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.606555 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.606561 | controller |       period = 900\n    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.606566 | controller |       1 ']'\n    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile
2026-03-14 09:56:26.606570 | controller |       = /etc/pki/tls/certs/mtls-ca.crt'\n    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.606574 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.606584 | controller |       \   logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:56:26.606589 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.606606 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.606612 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.606617 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.606621 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.606626 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.606630 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.606635 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.606639 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.606643 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.606648 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.606654 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.606658 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.606663 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.606667 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.606672 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.606676 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.606680 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.606685 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.606689 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.606693 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.606698 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.606702 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.606706 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.606711 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.606717 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.606721 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.606727 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.606731 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.606736 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.606740 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.606744 | controller |       period = 900\n    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.606749 | controller |       1 ']'\n    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled
2026-03-14 09:56:26.606753 | controller |       = true'\n    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.606757 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:56:26.606762 | controller |       09:50:07 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:56:26.606766 | controller |       transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.606770 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.606775 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.606785 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.606790 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.606794 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.606798 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.606803 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.606807 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.606811 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.606816 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.606820 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.606824 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.606829 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.606833 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.606837 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.606842 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.606846 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.606850 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.606855 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.606859 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.606863 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.606870 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.606874 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.606878 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.606883 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.606887 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.606891 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.606895 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.606900 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.606904 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.606909 | controller |       period = 900\n    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.606913 | controller |       1 ']'\n    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n
2026-03-14 09:56:26.606917 | controller |       \   logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.606925 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:56:26.606930 | controller |       09:50:07 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:56:26.606950 | controller |       transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.606954 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.606959 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.606963 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.606968 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.606972 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.606976 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.606986 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.606991 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.606995 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.607000 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.607004 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.607008 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.607013 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.607017 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.607021 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.607026 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.607030 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.607036 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.607041 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.607045 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.607050 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.607054 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.607058 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.607063 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.607067 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.607071 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.607076 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.607080 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.607084 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.607089 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.607093 | controller |       period = 900\n    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.607097 | controller |       1 ']'\n    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n
2026-03-14 09:56:26.607102 | controller |       \   logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.607106 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:56:26.607110 | controller |       09:50:07 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:56:26.607115 | controller |       transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.607119 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.607123 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.607128 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.607134 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.607139 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.607143 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.607147 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.607152 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.607156 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.607160 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.607165 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.607174 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.607180 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.607185 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.607189 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.607194 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.607198 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.607202 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.607207 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.607211 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.607217 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.607221 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.607226 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.607230 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.607234 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.607239 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.607243 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.607247 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.607252 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.607256 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.607260 | controller |       period = 900\n    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.607265 | controller |       1 ']'\n    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n
2026-03-14 09:56:26.607269 | controller |       \   logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.607273 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:56:26.607278 | controller |       09:50:08 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:56:26.607282 | controller |       transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.607286 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.607291 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.607295 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.607299 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.607304 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.607308 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.607313 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.607318 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.607323 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.607327 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.607331 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.607336 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.607340 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.607345 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.607349 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.607353 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.607363 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.607367 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.607372 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.607376 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.607381 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.607385 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.607389 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.607394 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.607398 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.607402 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.607408 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.607414 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.607420 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.607426 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.607431 | controller |       period = 900\n    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.607435 | controller |       1 ']'\n    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-applier-0
2026-03-14 09:56:26.607439 | controller |       ']'\n    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-applier
2026-03-14 09:56:26.607444 | controller |       config...'\n    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | Checking watcher-applier
2026-03-14 09:56:26.607448 | controller |       config...\n    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile
2026-03-14 09:56:26.607452 | controller |       = /etc/pki/tls/certs/mtls.crt '\n    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.607457 | controller |       | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.607461 | controller |       \   logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:56:26.607466 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.607472 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.607476 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.607481 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.607485 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.607489 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.607494 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.607498 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.607503 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.607507 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.607513 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.607519 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.607523 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.607528 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.607532 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.607536 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.607541 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.607545 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.607556 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.607560 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.607565 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:56:26.607569 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.607573 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.607578 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.607582 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.607586 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.607591 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.607606 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.607612 | controller |       period = 900\n    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.607617 | controller |       1 ']'\n    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile
2026-03-14 09:56:26.607621 | controller |       = /etc/pki/tls/private/mtls.key'\n    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.607626 | controller |       | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.607630 | controller |       \   logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:56:26.607636 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.607647 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.607652 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.607656 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.607661 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.607665 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.607669 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.607674 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.607678 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.607682 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.607687 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.607691 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.607696 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.607702 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.607708 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.607714 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.607718 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.607723 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.607727 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.607731 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.607736 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:56:26.607740 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.607744 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.607749 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.607753 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.607764 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.607769 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.607773 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.607778 | controller |       period = 900\n    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.607782 | controller |       1 ']'\n    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile
2026-03-14 09:56:26.607789 | controller |       = /etc/pki/tls/certs/mtls-ca.crt'\n    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.607794 | controller |       | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.607798 | controller |       \   logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:56:26.607805 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.607809 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.607814 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.607818 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.607822 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.607827 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.607831 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.607837 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.607842 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.607847 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.607852 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.607856 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.607860 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.607865 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.607869 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.607873 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.607878 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.607882 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.607886 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.607891 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.607895 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:56:26.607899 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.607904 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.607908 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.607912 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.607917 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.607921 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.607925 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.607932 | controller |       period = 900\n    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.607936 | controller |       1 ']'\n    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled
2026-03-14 09:56:26.607940 | controller |       = true'\n    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.607945 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:56:26.607955 | controller |       09:50:08 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:56:26.607959 | controller |       transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.607964 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.607968 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.607973 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.607977 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.607981 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.607989 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.607993 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.607998 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.608002 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.608006 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.608011 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.608015 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.608019 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.608024 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.608028 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.608032 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.608037 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.608041 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.608045 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.608050 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:56:26.608054 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.608058 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.608063 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.608067 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.608071 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.608077 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.608082 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.608086 | controller |       period = 900\n    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.608091 | controller |       1 ']'\n    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n
2026-03-14 09:56:26.608095 | controller |       \   logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.608099 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:56:26.608105 | controller |       09:50:09 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:56:26.608110 | controller |       transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.608114 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.608118 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.608123 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.608132 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.608137 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.608141 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.608145 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.608150 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.608154 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.608158 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.608163 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.608167 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.608171 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.608176 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.608180 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.608184 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.608189 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.608193 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.608197 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.608202 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:56:26.608206 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.608210 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.608215 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.608221 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.608225 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.608230 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.608234 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.608238 | controller |       period = 900\n    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.608243 | controller |       1 ']'\n    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n
2026-03-14 09:56:26.608247 | controller |       \   logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.608251 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:56:26.608256 | controller |       09:50:09 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:56:26.608261 | controller |       transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.608266 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.608270 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.608274 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.608279 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.608283 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.608287 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.608292 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.608296 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.608301 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.608305 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.608314 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.608319 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.608323 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.608328 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.608332 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.608336 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.608341 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.608345 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.608349 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.608354 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:56:26.608358 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.608364 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.608369 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.608373 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.608379 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.608383 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.608388 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.608392 | controller |       period = 900\n    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.608396 | controller |       1 ']'\n    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n
2026-03-14 09:56:26.608401 | controller |       \   logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.608405 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:56:26.608409 | controller |       09:50:09 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:56:26.608414 | controller |       transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.608418 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.608424 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.608429 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.608433 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.608437 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.608442 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.608446 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.608450 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.608455 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.608459 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.608465 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.608470 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.608476 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.608482 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.608489 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.608495 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.608501 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.608513 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.608518 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.608525 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:56:26.608530 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.608534 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.608539 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.608545 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.608551 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.608558 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.608567 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.608573 | controller |       period = 900\n    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.608578 | controller |       1 ']'\n    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.608582 | controller |       ']'\n    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-decision-engine
2026-03-14 09:56:26.608587 | controller |       config...'\n    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | Checking watcher-decision-engine
2026-03-14 09:56:26.608591 | controller |       config...\n    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile
2026-03-14 09:56:26.608610 | controller |       = /etc/pki/tls/certs/mtls.crt '\n    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.608617 | controller |       | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.608622 | controller |       \   logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:56:26.608626 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.608630 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.608635 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.608639 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.608644 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.608648 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.608652 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.608657 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.608661 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.608665 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.608669 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.608674 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.608678 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.608683 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.608689 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.608693 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.608698 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.608702 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.608706 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.608711 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.608717 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.608728 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.608732 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.608737 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.608741 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.608746 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.608750 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.608754 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.608759 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.608763 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.608767 | controller |       period = 900\n    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.608772 | controller |       1 ']'\n    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile
2026-03-14 09:56:26.608776 | controller |       = /etc/pki/tls/private/mtls.key'\n    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.608780 | controller |       | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.608785 | controller |       \   logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:56:26.608789 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.608794 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.608799 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.608803 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.608808 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.608812 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.608816 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.608821 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.608825 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.608831 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.608835 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.608840 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.608844 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.608848 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.608853 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.608857 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.608861 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.608866 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.608870 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.608874 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.608879 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.608883 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.608887 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.608892 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.608896 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.608900 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.608909 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.608916 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.608920 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.608925 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.608929 | controller |       period = 900\n    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.608933 | controller |       1 ']'\n    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile
2026-03-14 09:56:26.608938 | controller |       = /etc/pki/tls/certs/mtls-ca.crt'\n    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls
2026-03-14 09:56:26.608942 | controller |       | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.608947 | controller |       \   logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:56:26.608951 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.608955 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.608960 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.608964 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.608968 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.608974 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.608979 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.608983 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.608987 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.608992 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.608996 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.609000 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.609005 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.609012 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.609017 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.609021 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.609026 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.609037 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.609041 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.609046 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.609050 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.609054 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.609059 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.609063 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.609067 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.609072 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.609076 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.609080 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.609085 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.609089 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.609093 | controller |       period = 900\n    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.609103 | controller |       1 ']'\n    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled
2026-03-14 09:56:26.609108 | controller |       = true'\n    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:56:26.609112 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.609117 | controller |       \   logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:56:26.609121 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.609126 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.609132 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.609137 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.609141 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.609145 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.609150 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.609154 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.609158 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.609163 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.609167 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.609171 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.609176 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.609180 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.609184 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.609189 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.609193 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.609197 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.609202 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.609206 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.609210 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.609215 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.609219 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.609223 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.609228 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.609232 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.609236 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.609241 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.609245 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.609249 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.609254 | controller |       period = 900\n    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.609258 | controller |       1 ']'\n    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n
2026-03-14 09:56:26.609262 | controller |       \   logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:56:26.609267 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.609273 | controller |       \   logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:56:26.609284 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.609289 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.609293 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.609298 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.609302 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.609307 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.609313 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.609317 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.609322 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.609326 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.609330 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.609335 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.609339 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.609343 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.609348 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.609353 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.609357 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.609362 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.609366 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.609370 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.609375 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.609379 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.609383 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.609388 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.609392 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.609397 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.609401 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.609405 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.609410 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.609414 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.609420 | controller |       period = 900\n    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.609425 | controller |       1 ']'\n    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n
2026-03-14 09:56:26.609429 | controller |       \   logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:56:26.609433 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.609439 | controller |       \   logger.go:42: 09:50:11 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:56:26.609444 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.609448 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.609453 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.609457 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.609467 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.609472 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.609476 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.609480 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.609485 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.609489 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.609493 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.609498 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.609502 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.609507 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.609511 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.609515 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.609520 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.609524 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.609528 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.609533 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.609537 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.609541 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.609546 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.609550 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.609556 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.609565 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.609573 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.609579 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.609586 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.609592 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.609611 | controller |       period = 900\n    logger.go:42: 09:50:11 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.609615 | controller |       1 ']'\n    logger.go:42: 09:50:11 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n
2026-03-14 09:56:26.609620 | controller |       \   logger.go:42: 09:50:11 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:56:26.609624 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.609628 | controller |       \   logger.go:42: 09:50:11 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:56:26.609633 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.609637 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:56:26.609642 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.609646 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.609650 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.609655 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.609659 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.609663 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.609668 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.609678 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.609683 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.609687 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.609692 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.609696 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.609700 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.609705 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.609709 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.609713 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.609718 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.609722 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.609726 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.609731 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.609737 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:56:26.609742 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.609746 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:56:26.609750 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:56:26.609754 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.609761 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:56:26.609765 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.609769 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.609774 | controller |       period = 900\n    logger.go:42: 09:50:11 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:56:26.609778 | controller |       1 ']'\n    logger.go:42: 09:50:11 | watcher-tls/2-patch-mtls | test step completed
2026-03-14 09:56:26.609782 | controller |       2-patch-mtls\n    logger.go:42: 09:50:11 | watcher-tls/3-disable-podlevel-tls |
2026-03-14 09:56:26.609786 | controller |       starting test step 3-disable-podlevel-tls\n    logger.go:42: 09:50:11 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.609791 | controller |       | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json'
2026-03-14 09:56:26.609795 | controller |       -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/tls/api\", \"value\":{
2026-03-14 09:56:26.609799 | controller |       \"internal\": {}, \"public\": {} }}]'\n        ]\n    logger.go:42: 09:50:11 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.609804 | controller |       | watcher.watcher.openstack.org/watcher-kuttl patched\n    logger.go:42: 09:50:11
2026-03-14 09:56:26.609808 | controller |       | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:56:26.609812 | controller |       \       oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:56:26.609816 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n ${NAMESPACE}
2026-03-14 09:56:26.609821 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep
2026-03-14 09:56:26.609825 | controller |       watcher| awk '{print $1}')\n        [ $(oc get -n ${NAMESPACE} keystoneservice watcher
2026-03-14 09:56:26.609829 | controller |       -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE}
2026-03-14 09:56:26.609833 | controller |       watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n        # check that
2026-03-14 09:56:26.609838 | controller |       watcher internal endpoint does not use https\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:56:26.609844 | controller |       -- openstack endpoint list | grep infra-optim | grep internal  | [ $(grep -c https)
2026-03-14 09:56:26.609848 | controller |       == 0 ]\n        # check that watcher public endpoint does use https\n        oc
2026-03-14 09:56:26.609852 | controller |       exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim
2026-03-14 09:56:26.609857 | controller |       | grep public  | [ $(grep -c https) == 1 ]\n        # If we are running the container
2026-03-14 09:56:26.609861 | controller |       locally, skip following test\n        if [ \"$(oc get pods -n openstack-operators
2026-03-14 09:56:26.609865 | controller |       -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit
2026-03-14 09:56:26.609869 | controller |       0\n        fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:56:26.609879 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:56:26.609883 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:56:26.609890 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:56:26.609894 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:56:26.609898 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:56:26.609902 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:56:26.609907 | controller |       \       ]\n    logger.go:42: 09:50:11 | watcher-tls/3-disable-podlevel-tls | + oc
2026-03-14 09:56:26.609911 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.609915 | controller |       -c Name -c Type\n    logger.go:42: 09:50:11 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.609920 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:50:13 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.609924 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:50:13 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.609928 | controller |       | ++ grep watcher\n    logger.go:42: 09:50:13 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.609932 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.609937 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:50:13 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.609941 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:50:16 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.609945 | controller |       | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa\n    logger.go:42: 09:50:16 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.609949 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.609954 | controller |       \   logger.go:42: 09:50:16 | watcher-tls/3-disable-podlevel-tls | + '[' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.609958 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa ']'\n    logger.go:42: 09:50:16 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.609962 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:56:26.609966 | controller |       \   logger.go:42: 09:50:16 | watcher-tls/3-disable-podlevel-tls | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.609973 | controller |       ']'\n    logger.go:42: 09:50:16 | watcher-tls/3-disable-podlevel-tls | + grep internal\n
2026-03-14 09:56:26.609977 | controller |       \   logger.go:42: 09:50:16 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n
2026-03-14 09:56:26.609982 | controller |       \   logger.go:42: 09:50:16 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n
2026-03-14 09:56:26.609986 | controller |       \   logger.go:42: 09:50:16 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.609990 | controller |       openstackclient -- openstack endpoint list\n    logger.go:42: 09:50:18 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.609994 | controller |       | + '[' 1 == 0 ']'\n    logger.go:42: 09:50:19 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.609999 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:56:26.610003 | controller |       -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n
2026-03-14 09:56:26.610007 | controller |       \       SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.610011 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n        [
2026-03-14 09:56:26.610016 | controller |       $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.610020 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl
2026-03-14 09:56:26.610024 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # check that watcher internal endpoint
2026-03-14 09:56:26.610030 | controller |       does not use https\n        oc exec -n ${NAMESPACE} openstackclient -- openstack
2026-03-14 09:56:26.610034 | controller |       endpoint list | grep infra-optim | grep internal  | [ $(grep -c https) == 0 ]\n
2026-03-14 09:56:26.610040 | controller |       \       # check that watcher public endpoint does use https\n        oc exec -n
2026-03-14 09:56:26.610045 | controller |       ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep
2026-03-14 09:56:26.610049 | controller |       public  | [ $(grep -c https) == 1 ]\n        # If we are running the container locally,
2026-03-14 09:56:26.610053 | controller |       skip following test\n        if [ \"$(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.610058 | controller |       -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit 0\n
2026-03-14 09:56:26.610062 | controller |       \       fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:56:26.610071 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:56:26.610076 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:56:26.610080 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:56:26.610084 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:56:26.610089 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:56:26.610093 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:56:26.610097 | controller |       \       ]\n    logger.go:42: 09:50:19 | watcher-tls/3-disable-podlevel-tls | + oc
2026-03-14 09:56:26.610101 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.610106 | controller |       -c Name -c Type\n    logger.go:42: 09:50:19 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610110 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:50:21 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610114 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:50:21 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610119 | controller |       | ++ grep watcher\n    logger.go:42: 09:50:21 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610123 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:50:21 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610127 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.610131 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:50:23 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610136 | controller |       | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa\n    logger.go:42: 09:50:23 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610140 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.610144 | controller |       \   logger.go:42: 09:50:24 | watcher-tls/3-disable-podlevel-tls | + '[' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.610148 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa ']'\n    logger.go:42: 09:50:24 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610153 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:56:26.610160 | controller |       \   logger.go:42: 09:50:24 | watcher-tls/3-disable-podlevel-tls | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.610165 | controller |       ']'\n    logger.go:42: 09:50:24 | watcher-tls/3-disable-podlevel-tls | + grep internal\n
2026-03-14 09:56:26.610169 | controller |       \   logger.go:42: 09:50:24 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n
2026-03-14 09:56:26.610173 | controller |       \   logger.go:42: 09:50:24 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n
2026-03-14 09:56:26.610179 | controller |       \   logger.go:42: 09:50:24 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.610183 | controller |       openstackclient -- openstack endpoint list\n    logger.go:42: 09:50:26 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610188 | controller |       | + '[' 1 == 0 ']'\n    logger.go:42: 09:50:27 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610192 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:56:26.610196 | controller |       -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n
2026-03-14 09:56:26.610200 | controller |       \       SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.610205 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n        [
2026-03-14 09:56:26.610209 | controller |       $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.610213 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl
2026-03-14 09:56:26.610217 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # check that watcher internal endpoint
2026-03-14 09:56:26.610222 | controller |       does not use https\n        oc exec -n ${NAMESPACE} openstackclient -- openstack
2026-03-14 09:56:26.610232 | controller |       endpoint list | grep infra-optim | grep internal  | [ $(grep -c https) == 0 ]\n
2026-03-14 09:56:26.610237 | controller |       \       # check that watcher public endpoint does use https\n        oc exec -n
2026-03-14 09:56:26.610243 | controller |       ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep
2026-03-14 09:56:26.610247 | controller |       public  | [ $(grep -c https) == 1 ]\n        # If we are running the container locally,
2026-03-14 09:56:26.610252 | controller |       skip following test\n        if [ \"$(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.610256 | controller |       -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit 0\n
2026-03-14 09:56:26.610260 | controller |       \       fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:56:26.610264 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:56:26.610274 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:56:26.610279 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:56:26.610283 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:56:26.610287 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:56:26.610292 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:56:26.610298 | controller |       \       ]\n    logger.go:42: 09:50:27 | watcher-tls/3-disable-podlevel-tls | + oc
2026-03-14 09:56:26.610302 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.610307 | controller |       -c Name -c Type\n    logger.go:42: 09:50:27 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610311 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:50:29 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610315 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:50:29 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610319 | controller |       | ++ grep watcher\n    logger.go:42: 09:50:29 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610323 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:50:29 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610328 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.610332 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:50:31 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610336 | controller |       | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa\n    logger.go:42: 09:50:31 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610342 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.610347 | controller |       \   logger.go:42: 09:50:31 | watcher-tls/3-disable-podlevel-tls | + '[' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.610352 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa ']'\n    logger.go:42: 09:50:31 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610357 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:56:26.610361 | controller |       \   logger.go:42: 09:50:31 | watcher-tls/3-disable-podlevel-tls | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.610365 | controller |       ']'\n    logger.go:42: 09:50:31 | watcher-tls/3-disable-podlevel-tls | + grep internal\n
2026-03-14 09:56:26.610370 | controller |       \   logger.go:42: 09:50:31 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n
2026-03-14 09:56:26.610374 | controller |       \   logger.go:42: 09:50:31 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.610378 | controller |       openstackclient -- openstack endpoint list\n    logger.go:42: 09:50:31 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610383 | controller |       | ++ grep -c https\n    logger.go:42: 09:50:33 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610387 | controller |       | + '[' 1 == 0 ']'\n    logger.go:42: 09:50:34 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610391 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:56:26.610395 | controller |       -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n
2026-03-14 09:56:26.610400 | controller |       \       SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.610404 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n        [
2026-03-14 09:56:26.610408 | controller |       $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.610412 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl
2026-03-14 09:56:26.610417 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # check that watcher internal endpoint
2026-03-14 09:56:26.610421 | controller |       does not use https\n        oc exec -n ${NAMESPACE} openstackclient -- openstack
2026-03-14 09:56:26.610425 | controller |       endpoint list | grep infra-optim | grep internal  | [ $(grep -c https) == 0 ]\n
2026-03-14 09:56:26.610429 | controller |       \       # check that watcher public endpoint does use https\n        oc exec -n
2026-03-14 09:56:26.610434 | controller |       ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep
2026-03-14 09:56:26.610439 | controller |       public  | [ $(grep -c https) == 1 ]\n        # If we are running the container locally,
2026-03-14 09:56:26.610444 | controller |       skip following test\n        if [ \"$(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.610448 | controller |       -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit 0\n
2026-03-14 09:56:26.610452 | controller |       \       fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:56:26.610457 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:56:26.610466 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:56:26.610471 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:56:26.610475 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:56:26.610479 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:56:26.610484 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:56:26.610488 | controller |       \       ]\n    logger.go:42: 09:50:34 | watcher-tls/3-disable-podlevel-tls | + oc
2026-03-14 09:56:26.610494 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.610498 | controller |       -c Name -c Type\n    logger.go:42: 09:50:34 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610502 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:50:36 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610507 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:50:36 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610511 | controller |       | ++ grep watcher\n    logger.go:42: 09:50:36 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610515 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.610520 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:50:36 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610524 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:50:38 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610528 | controller |       | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa\n    logger.go:42: 09:50:38 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610532 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.610537 | controller |       \   logger.go:42: 09:50:39 | watcher-tls/3-disable-podlevel-tls | + '[' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.610541 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa ']'\n    logger.go:42: 09:50:39 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610545 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:56:26.610549 | controller |       \   logger.go:42: 09:50:39 | watcher-tls/3-disable-podlevel-tls | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.610554 | controller |       ']'\n    logger.go:42: 09:50:39 | watcher-tls/3-disable-podlevel-tls | + grep internal\n
2026-03-14 09:56:26.610558 | controller |       \   logger.go:42: 09:50:39 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n
2026-03-14 09:56:26.610562 | controller |       \   logger.go:42: 09:50:39 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n
2026-03-14 09:56:26.610567 | controller |       \   logger.go:42: 09:50:39 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.610571 | controller |       openstackclient -- openstack endpoint list\n    logger.go:42: 09:50:41 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610575 | controller |       | + '[' 1 == 0 ']'\n    logger.go:42: 09:50:42 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610579 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:56:26.610584 | controller |       -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n
2026-03-14 09:56:26.610588 | controller |       \       SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.610592 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n        [
2026-03-14 09:56:26.610610 | controller |       $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.610614 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl
2026-03-14 09:56:26.610618 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # check that watcher internal endpoint
2026-03-14 09:56:26.610623 | controller |       does not use https\n        oc exec -n ${NAMESPACE} openstackclient -- openstack
2026-03-14 09:56:26.610627 | controller |       endpoint list | grep infra-optim | grep internal  | [ $(grep -c https) == 0 ]\n
2026-03-14 09:56:26.610631 | controller |       \       # check that watcher public endpoint does use https\n        oc exec -n
2026-03-14 09:56:26.610635 | controller |       ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep
2026-03-14 09:56:26.610640 | controller |       public  | [ $(grep -c https) == 1 ]\n        # If we are running the container locally,
2026-03-14 09:56:26.610651 | controller |       skip following test\n        if [ \"$(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.610656 | controller |       -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit 0\n
2026-03-14 09:56:26.610660 | controller |       \       fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:56:26.610665 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:56:26.610671 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:56:26.610684 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:56:26.610690 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:56:26.610696 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:56:26.610703 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:56:26.610708 | controller |       \       ]\n    logger.go:42: 09:50:42 | watcher-tls/3-disable-podlevel-tls | + oc
2026-03-14 09:56:26.610712 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.610717 | controller |       -c Name -c Type\n    logger.go:42: 09:50:42 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610721 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:50:44 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610725 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:50:44 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610729 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.610734 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:50:44 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610738 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:50:44 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610742 | controller |       | ++ grep watcher\n    logger.go:42: 09:50:46 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610747 | controller |       | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa\n    logger.go:42: 09:50:46 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610751 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.610755 | controller |       \   logger.go:42: 09:50:46 | watcher-tls/3-disable-podlevel-tls | + '[' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.610760 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa ']'\n    logger.go:42: 09:50:46 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610764 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:56:26.610768 | controller |       \   logger.go:42: 09:50:47 | watcher-tls/3-disable-podlevel-tls | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.610772 | controller |       ']'\n    logger.go:42: 09:50:47 | watcher-tls/3-disable-podlevel-tls | + grep internal\n
2026-03-14 09:56:26.610777 | controller |       \   logger.go:42: 09:50:47 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n
2026-03-14 09:56:26.610781 | controller |       \   logger.go:42: 09:50:47 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.610785 | controller |       openstackclient -- openstack endpoint list\n    logger.go:42: 09:50:47 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610790 | controller |       | ++ grep -c https\n    logger.go:42: 09:50:49 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610794 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:50:49 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610798 | controller |       | + grep public\n    logger.go:42: 09:50:49 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610802 | controller |       | + grep infra-optim\n    logger.go:42: 09:50:49 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610809 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list\n
2026-03-14 09:56:26.610813 | controller |       \   logger.go:42: 09:50:49 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n
2026-03-14 09:56:26.610817 | controller |       \   logger.go:42: 09:50:51 | watcher-tls/3-disable-podlevel-tls | + '[' 1 == 1 ']'\n
2026-03-14 09:56:26.610822 | controller |       \   logger.go:42: 09:50:51 | watcher-tls/3-disable-podlevel-tls | ++ oc get pods
2026-03-14 09:56:26.610826 | controller |       -n openstack-operators -o name -l openstack.org/operator-name=watcher\n    logger.go:42:
2026-03-14 09:56:26.610830 | controller |       09:50:51 | watcher-tls/3-disable-podlevel-tls | + '[' '' == '' ']'\n    logger.go:42:
2026-03-14 09:56:26.610834 | controller |       09:50:51 | watcher-tls/3-disable-podlevel-tls | + exit 0\n    logger.go:42: 09:50:52
2026-03-14 09:56:26.610839 | controller |       | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:56:26.610843 | controller |       \       oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:56:26.610847 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n ${NAMESPACE}
2026-03-14 09:56:26.610852 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep
2026-03-14 09:56:26.610858 | controller |       watcher| awk '{print $1}')\n        [ $(oc get -n ${NAMESPACE} keystoneservice watcher
2026-03-14 09:56:26.610862 | controller |       -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE}
2026-03-14 09:56:26.610867 | controller |       watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n        # check that
2026-03-14 09:56:26.610871 | controller |       watcher internal endpoint does not use https\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:56:26.610881 | controller |       -- openstack endpoint list | grep infra-optim | grep internal  | [ $(grep -c https)
2026-03-14 09:56:26.610886 | controller |       == 0 ]\n        # check that watcher public endpoint does use https\n        oc
2026-03-14 09:56:26.610890 | controller |       exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim
2026-03-14 09:56:26.610894 | controller |       | grep public  | [ $(grep -c https) == 1 ]\n        # If we are running the container
2026-03-14 09:56:26.610898 | controller |       locally, skip following test\n        if [ \"$(oc get pods -n openstack-operators
2026-03-14 09:56:26.610903 | controller |       -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit
2026-03-14 09:56:26.610907 | controller |       0\n        fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:56:26.610911 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:56:26.610915 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:56:26.610920 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:56:26.610924 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:56:26.610928 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:56:26.610932 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:56:26.610954 | controller |       \       ]\n    logger.go:42: 09:50:52 | watcher-tls/3-disable-podlevel-tls | + oc
2026-03-14 09:56:26.610959 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.610963 | controller |       -c Name -c Type\n    logger.go:42: 09:50:52 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610968 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:50:54 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610972 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:50:54 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610976 | controller |       | ++ grep watcher\n    logger.go:42: 09:50:54 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610982 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:50:54 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610987 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.610991 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:50:56 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.610995 | controller |       | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa\n    logger.go:42: 09:50:56 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.611000 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.611006 | controller |       \   logger.go:42: 09:50:56 | watcher-tls/3-disable-podlevel-tls | + '[' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.611010 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa ']'\n    logger.go:42: 09:50:56 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.611015 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:56:26.611019 | controller |       \   logger.go:42: 09:50:57 | watcher-tls/3-disable-podlevel-tls | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.611023 | controller |       ']'\n    logger.go:42: 09:50:57 | watcher-tls/3-disable-podlevel-tls | + oc exec
2026-03-14 09:56:26.611027 | controller |       -n watcher-kuttl-default openstackclient -- openstack endpoint list\n    logger.go:42:
2026-03-14 09:56:26.611032 | controller |       09:50:57 | watcher-tls/3-disable-podlevel-tls | + grep internal\n    logger.go:42:
2026-03-14 09:56:26.611036 | controller |       09:50:57 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n    logger.go:42:
2026-03-14 09:56:26.611040 | controller |       09:50:57 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n    logger.go:42:
2026-03-14 09:56:26.611045 | controller |       09:50:59 | watcher-tls/3-disable-podlevel-tls | + '[' 0 == 0 ']'\n    logger.go:42:
2026-03-14 09:56:26.611049 | controller |       09:50:59 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.611053 | controller |       openstackclient -- openstack endpoint list\n    logger.go:42: 09:50:59 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.611057 | controller |       | + grep public\n    logger.go:42: 09:50:59 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.611062 | controller |       | ++ grep -c https\n    logger.go:42: 09:50:59 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.611066 | controller |       | + grep infra-optim\n    logger.go:42: 09:51:01 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.611072 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:51:01 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:56:26.611076 | controller |       | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n
2026-03-14 09:56:26.611080 | controller |       \   logger.go:42: 09:51:01 | watcher-tls/3-disable-podlevel-tls | + '[' '' == ''
2026-03-14 09:56:26.611085 | controller |       ']'\n    logger.go:42: 09:51:01 | watcher-tls/3-disable-podlevel-tls | + exit 0\n
2026-03-14 09:56:26.611095 | controller |       \   logger.go:42: 09:51:01 | watcher-tls/3-disable-podlevel-tls | test step completed
2026-03-14 09:56:26.611099 | controller |       3-disable-podlevel-tls\n    logger.go:42: 09:51:01 | watcher-tls/4-deploy-without-route
2026-03-14 09:56:26.611103 | controller |       | starting test step 4-deploy-without-route\n    logger.go:42: 09:51:01 | watcher-tls/4-deploy-without-route
2026-03-14 09:56:26.611108 | controller |       | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json'
2026-03-14 09:56:26.611112 | controller |       -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/override\", \"value\":{\"service\":
2026-03-14 09:56:26.611116 | controller |       { \"internal\": {}, \"public\": { \"metadata\": { \"annotations\": { \"metallb.universe.tf/address-pool\":
2026-03-14 09:56:26.611121 | controller |       \"ctlplane\", \"metallb.universe.tf/allow-shared-ip\": \"ctlplane\" } }, \"spec\":
2026-03-14 09:56:26.611125 | controller |       { \"type\": \"LoadBalancer\" } } } }}]'\n        ]\n    logger.go:42: 09:51:01 |
2026-03-14 09:56:26.611132 | controller |       watcher-tls/4-deploy-without-route | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:56:26.611137 | controller |       patched\n    logger.go:42: 09:51:01 | watcher-tls/4-deploy-without-route | running
2026-03-14 09:56:26.611141 | controller |       command: [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:56:26.611145 | controller |       -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n
2026-03-14 09:56:26.611149 | controller |       \       SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.611154 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n        [
2026-03-14 09:56:26.611158 | controller |       $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.611162 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl
2026-03-14 09:56:26.611166 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # If we are running the container
2026-03-14 09:56:26.611171 | controller |       locally, skip following test\n        if [ \"$(oc get pods -n openstack-operators
2026-03-14 09:56:26.611175 | controller |       -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit
2026-03-14 09:56:26.611179 | controller |       0\n        fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:56:26.611183 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:56:26.611188 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:56:26.611192 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:56:26.611196 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:56:26.611200 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:56:26.611205 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:56:26.611209 | controller |       \       ]\n    logger.go:42: 09:51:01 | watcher-tls/4-deploy-without-route | + oc
2026-03-14 09:56:26.611213 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.611218 | controller |       -c Name -c Type\n    logger.go:42: 09:51:01 | watcher-tls/4-deploy-without-route
2026-03-14 09:56:26.611222 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:51:03 | watcher-tls/4-deploy-without-route
2026-03-14 09:56:26.611226 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:51:03 | watcher-tls/4-deploy-without-route
2026-03-14 09:56:26.611230 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.611234 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:51:03 | watcher-tls/4-deploy-without-route
2026-03-14 09:56:26.611241 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:51:03 | watcher-tls/4-deploy-without-route
2026-03-14 09:56:26.611245 | controller |       | ++ grep watcher\n    logger.go:42: 09:51:05 | watcher-tls/4-deploy-without-route
2026-03-14 09:56:26.611249 | controller |       | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa\n    logger.go:42: 09:51:05 | watcher-tls/4-deploy-without-route
2026-03-14 09:56:26.611254 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.611258 | controller |       \   logger.go:42: 09:51:06 | watcher-tls/4-deploy-without-route | + '[' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.611262 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa ']'\n    logger.go:42: 09:51:06 | watcher-tls/4-deploy-without-route
2026-03-14 09:56:26.611267 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:56:26.611271 | controller |       \   logger.go:42: 09:51:06 | watcher-tls/4-deploy-without-route | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.611277 | controller |       ']'\n    logger.go:42: 09:51:06 | watcher-tls/4-deploy-without-route | ++ oc get
2026-03-14 09:56:26.611286 | controller |       pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n    logger.go:42:
2026-03-14 09:56:26.611290 | controller |       09:51:06 | watcher-tls/4-deploy-without-route | + '[' '' == '' ']'\n    logger.go:42:
2026-03-14 09:56:26.611295 | controller |       09:51:06 | watcher-tls/4-deploy-without-route | + exit 0\n    logger.go:42: 09:51:07
2026-03-14 09:56:26.611299 | controller |       | watcher-tls/4-deploy-without-route | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:56:26.611304 | controller |       \       oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:56:26.611308 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n ${NAMESPACE}
2026-03-14 09:56:26.611312 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep
2026-03-14 09:56:26.611316 | controller |       watcher| awk '{print $1}')\n        [ $(oc get -n ${NAMESPACE} keystoneservice watcher
2026-03-14 09:56:26.611321 | controller |       -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE}
2026-03-14 09:56:26.611325 | controller |       watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n        # If we are
2026-03-14 09:56:26.611329 | controller |       running the container locally, skip following test\n        if [ \"$(oc get pods
2026-03-14 09:56:26.611333 | controller |       -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\"
2026-03-14 09:56:26.611338 | controller |       ]; then\n            exit 0\n        fi\n        env_variables=$(oc set env $(oc
2026-03-14 09:56:26.611342 | controller |       get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:56:26.611346 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:56:26.611350 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:56:26.611355 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:56:26.611359 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:56:26.611363 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:56:26.611368 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:51:07 | watcher-tls/4-deploy-without-route
2026-03-14 09:56:26.611372 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.611376 | controller |       value -c Name -c Type\n    logger.go:42: 09:51:07 | watcher-tls/4-deploy-without-route
2026-03-14 09:56:26.611380 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:51:09 | watcher-tls/4-deploy-without-route
2026-03-14 09:56:26.611385 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:51:09 | watcher-tls/4-deploy-without-route
2026-03-14 09:56:26.611389 | controller |       | ++ grep watcher\n    logger.go:42: 09:51:09 | watcher-tls/4-deploy-without-route
2026-03-14 09:56:26.611393 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:51:09 | watcher-tls/4-deploy-without-route
2026-03-14 09:56:26.611398 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.611402 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:51:11 | watcher-tls/4-deploy-without-route
2026-03-14 09:56:26.611406 | controller |       | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa\n    logger.go:42: 09:51:11 | watcher-tls/4-deploy-without-route
2026-03-14 09:56:26.611410 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.611415 | controller |       \   logger.go:42: 09:51:11 | watcher-tls/4-deploy-without-route | + '[' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.611419 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa ']'\n    logger.go:42: 09:51:11 | watcher-tls/4-deploy-without-route
2026-03-14 09:56:26.611425 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:56:26.611429 | controller |       \   logger.go:42: 09:51:11 | watcher-tls/4-deploy-without-route | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.611434 | controller |       ']'\n    logger.go:42: 09:51:11 | watcher-tls/4-deploy-without-route | ++ oc get
2026-03-14 09:56:26.611438 | controller |       pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n    logger.go:42:
2026-03-14 09:56:26.611442 | controller |       09:51:12 | watcher-tls/4-deploy-without-route | + '[' '' == '' ']'\n    logger.go:42:
2026-03-14 09:56:26.611447 | controller |       09:51:12 | watcher-tls/4-deploy-without-route | + exit 0\n    logger.go:42: 09:51:12
2026-03-14 09:56:26.611451 | controller |       | watcher-tls/4-deploy-without-route | test step completed 4-deploy-without-route\n
2026-03-14 09:56:26.611455 | controller |       \   logger.go:42: 09:51:12 | watcher-tls/5-disable-tls | starting test step 5-disable-tls\n
2026-03-14 09:56:26.611459 | controller |       \   logger.go:42: 09:51:12 | watcher-tls/5-disable-tls | running command: [sh -c
2026-03-14 09:56:26.611464 | controller |       oc patch watcher -n $NAMESPACE watcher-kuttl --type='json' -p='[{\"op\": \"replace\",
2026-03-14 09:56:26.611473 | controller |       \"path\": \"/spec/apiServiceTemplate/override\", \"value\":{}}]'\n        ]\n    logger.go:42:
2026-03-14 09:56:26.611478 | controller |       09:51:12 | watcher-tls/5-disable-tls | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:56:26.611484 | controller |       patched\n    logger.go:42: 09:51:12 | watcher-tls/5-disable-tls | running command:
2026-03-14 09:56:26.611489 | controller |       [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient -- openstack
2026-03-14 09:56:26.611493 | controller |       service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc
2026-03-14 09:56:26.611498 | controller |       exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.611502 | controller |       -c Type -c ID | grep watcher| awk '{print $1}')\n        [ $(oc get -n ${NAMESPACE}
2026-03-14 09:56:26.611506 | controller |       keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [
2026-03-14 09:56:26.611510 | controller |       -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\"
2026-03-14 09:56:26.611515 | controller |       ]\n        # If we are running the container locally, skip following test\n        if
2026-03-14 09:56:26.611519 | controller |       [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:56:26.611523 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:56:26.611527 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:56:26.611532 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:56:26.611536 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:56:26.611540 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:56:26.611545 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:56:26.611549 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:56:26.611553 | controller |       variables found.\"\n        fi\n        # check that no watcher endpoint uses https\n
2026-03-14 09:56:26.611557 | controller |       \       oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.611562 | controller |       infra-optim | [ $(grep -c https) == 0 ]\n        ]\n    logger.go:42: 09:51:12 |
2026-03-14 09:56:26.611566 | controller |       watcher-tls/5-disable-tls | + oc exec -n watcher-kuttl-default openstackclient --
2026-03-14 09:56:26.611574 | controller |       openstack service list -f value -c Name -c Type\n    logger.go:42: 09:51:12 | watcher-tls/5-disable-tls
2026-03-14 09:56:26.611578 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:51:14 | watcher-tls/5-disable-tls
2026-03-14 09:56:26.611582 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:51:14 | watcher-tls/5-disable-tls | ++
2026-03-14 09:56:26.611587 | controller |       grep watcher\n    logger.go:42: 09:51:14 | watcher-tls/5-disable-tls | ++ awk '{print
2026-03-14 09:56:26.611591 | controller |       $1}'\n    logger.go:42: 09:51:14 | watcher-tls/5-disable-tls | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.611613 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID\n    logger.go:42:
2026-03-14 09:56:26.611620 | controller |       09:51:16 | watcher-tls/5-disable-tls | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa\n
2026-03-14 09:56:26.611624 | controller |       \   logger.go:42: 09:51:16 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.611629 | controller |       keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n    logger.go:42: 09:51:16
2026-03-14 09:56:26.611633 | controller |       | watcher-tls/5-disable-tls | + '[' bf6522b7dcd544e8aef389dfe89ae1fa == bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.611637 | controller |       ']'\n    logger.go:42: 09:51:16 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.611642 | controller |       watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n    logger.go:42: 09:51:16
2026-03-14 09:56:26.611646 | controller |       | watcher-tls/5-disable-tls | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.611650 | controller |       ']'\n    logger.go:42: 09:51:16 | watcher-tls/5-disable-tls | ++ oc get pods -n
2026-03-14 09:56:26.611655 | controller |       openstack-operators -o name -l openstack.org/operator-name=watcher\n    logger.go:42:
2026-03-14 09:56:26.611659 | controller |       09:51:16 | watcher-tls/5-disable-tls | + '[' '' == '' ']'\n    logger.go:42: 09:51:16
2026-03-14 09:56:26.611663 | controller |       | watcher-tls/5-disable-tls | + exit 0\n    logger.go:42: 09:51:17 | watcher-tls/5-disable-tls
2026-03-14 09:56:26.611668 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:56:26.611672 | controller |       -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n
2026-03-14 09:56:26.611676 | controller |       \       SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.611681 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n        [
2026-03-14 09:56:26.611690 | controller |       $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.611695 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl
2026-03-14 09:56:26.611700 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # If we are running the container
2026-03-14 09:56:26.611706 | controller |       locally, skip following test\n        if [ \"$(oc get pods -n openstack-operators
2026-03-14 09:56:26.611710 | controller |       -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit
2026-03-14 09:56:26.611715 | controller |       0\n        fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:56:26.611719 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:56:26.611723 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:56:26.611728 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:56:26.611732 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:56:26.611736 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:56:26.611741 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:56:26.611747 | controller |       \       # check that no watcher endpoint uses https\n        oc exec -n ${NAMESPACE}
2026-03-14 09:56:26.611751 | controller |       openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https)
2026-03-14 09:56:26.611755 | controller |       == 0 ]\n        ]\n    logger.go:42: 09:51:17 | watcher-tls/5-disable-tls | + oc
2026-03-14 09:56:26.611759 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.611764 | controller |       -c Name -c Type\n    logger.go:42: 09:51:17 | watcher-tls/5-disable-tls | ++ grep
2026-03-14 09:56:26.611768 | controller |       -c '^watcher'\n    logger.go:42: 09:51:19 | watcher-tls/5-disable-tls | + '[' 1
2026-03-14 09:56:26.611772 | controller |       == 1 ']'\n    logger.go:42: 09:51:19 | watcher-tls/5-disable-tls | ++ oc exec -n
2026-03-14 09:56:26.611777 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.611781 | controller |       -c Type -c ID\n    logger.go:42: 09:51:19 | watcher-tls/5-disable-tls | ++ awk '{print
2026-03-14 09:56:26.611785 | controller |       $1}'\n    logger.go:42: 09:51:19 | watcher-tls/5-disable-tls | ++ grep watcher\n
2026-03-14 09:56:26.611789 | controller |       \   logger.go:42: 09:51:21 | watcher-tls/5-disable-tls | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa\n
2026-03-14 09:56:26.611794 | controller |       \   logger.go:42: 09:51:21 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.611798 | controller |       keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n    logger.go:42: 09:51:21
2026-03-14 09:56:26.611802 | controller |       | watcher-tls/5-disable-tls | + '[' bf6522b7dcd544e8aef389dfe89ae1fa == bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.611806 | controller |       ']'\n    logger.go:42: 09:51:21 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.611811 | controller |       watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n    logger.go:42: 09:51:22
2026-03-14 09:56:26.611815 | controller |       | watcher-tls/5-disable-tls | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.611819 | controller |       ']'\n    logger.go:42: 09:51:22 | watcher-tls/5-disable-tls | ++ oc get pods -n
2026-03-14 09:56:26.611823 | controller |       openstack-operators -o name -l openstack.org/operator-name=watcher\n    logger.go:42:
2026-03-14 09:56:26.611828 | controller |       09:51:22 | watcher-tls/5-disable-tls | + '[' '' == '' ']'\n    logger.go:42: 09:51:22
2026-03-14 09:56:26.611832 | controller |       | watcher-tls/5-disable-tls | + exit 0\n    logger.go:42: 09:51:22 | watcher-tls/5-disable-tls
2026-03-14 09:56:26.611836 | controller |       | test step completed 5-disable-tls\n    logger.go:42: 09:51:22 | watcher-tls/6-cleanup-watcher
2026-03-14 09:56:26.611840 | controller |       | starting test step 6-cleanup-watcher\n    logger.go:42: 09:51:22 | watcher-tls/6-cleanup-watcher
2026-03-14 09:56:26.611845 | controller |       | running command: [sh -c set -ex\n        oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.611849 | controller |       -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0
2026-03-14 09:56:26.611853 | controller |       ]\n        ]\n    logger.go:42: 09:51:22 | watcher-tls/6-cleanup-watcher | + oc
2026-03-14 09:56:26.611858 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.611862 | controller |       -c Name -c Type\n    logger.go:42: 09:51:22 | watcher-tls/6-cleanup-watcher | ++
2026-03-14 09:56:26.611866 | controller |       grep -c '^watcher'\n    logger.go:42: 09:51:24 | watcher-tls/6-cleanup-watcher |
2026-03-14 09:56:26.611870 | controller |       + '[' 1 == 0 ']'\n    logger.go:42: 09:51:26 | watcher-tls/6-cleanup-watcher | running
2026-03-14 09:56:26.611875 | controller |       command: [sh -c set -ex\n        oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.611879 | controller |       -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0
2026-03-14 09:56:26.611888 | controller |       ]\n        ]\n    logger.go:42: 09:51:26 | watcher-tls/6-cleanup-watcher | + oc
2026-03-14 09:56:26.611894 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.611900 | controller |       -c Name -c Type\n    logger.go:42: 09:51:26 | watcher-tls/6-cleanup-watcher | ++
2026-03-14 09:56:26.611905 | controller |       grep -c '^watcher'\n    logger.go:42: 09:51:28 | watcher-tls/6-cleanup-watcher |
2026-03-14 09:56:26.611909 | controller |       + '[' 0 == 0 ']'\n    logger.go:42: 09:51:28 | watcher-tls/6-cleanup-watcher | test
2026-03-14 09:56:26.611913 | controller |       step completed 6-cleanup-watcher\n    logger.go:42: 09:51:28 | watcher-tls/7-cleanup-certs
2026-03-14 09:56:26.611918 | controller |       | starting test step 7-cleanup-certs\n    logger.go:42: 09:51:28 | watcher-tls/7-cleanup-certs
2026-03-14 09:56:26.611922 | controller |       | test step completed 7-cleanup-certs\n    logger.go:42: 09:51:28 | watcher-tls
2026-03-14 09:56:26.611926 | controller |       | skipping kubernetes event logging\n=== CONT  kuttl/harness/watcher-rmquser\n    logger.go:42:
2026-03-14 09:56:26.611931 | controller |       09:51:28 | watcher-rmquser | Skipping creation of user-supplied namespace: watcher-kuttl-default\n
2026-03-14 09:56:26.611935 | controller |       \   logger.go:42: 09:51:28 | watcher-rmquser/0-cleanup-watcher | starting test step
2026-03-14 09:56:26.611939 | controller |       0-cleanup-watcher\n    logger.go:42: 09:51:28 | watcher-rmquser/0-cleanup-watcher
2026-03-14 09:56:26.611943 | controller |       | test step completed 0-cleanup-watcher\n    logger.go:42: 09:51:28 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.611948 | controller |       | starting test step 1-deploy\n    logger.go:42: 09:51:28 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.611952 | controller |       | Watcher:watcher-kuttl-default/watcher-kuttl created\n    logger.go:42: 09:51:28
2026-03-14 09:56:26.611956 | controller |       | watcher-rmquser/1-deploy | running command: [sh -c set -euxo pipefail\n        \n
2026-03-14 09:56:26.611961 | controller |       \       # Wait for Watcher to be Ready\n        kubectl wait --for=condition=Ready
2026-03-14 09:56:26.611965 | controller |       watcher/watcher-kuttl -n $NAMESPACE --timeout=300s\n        \n        # Verify WatcherNotificationTransportURLReady
2026-03-14 09:56:26.611969 | controller |       condition exists and is True\n        kubectl get watcher watcher-kuttl -n $NAMESPACE
2026-03-14 09:56:26.611975 | controller |       -o jsonpath='{.status.conditions[?(@.type==\"WatcherNotificationTransportURLReady\")].status}'
2026-03-14 09:56:26.611979 | controller |       | grep -q \"True\"\n        echo \"WatcherNotificationTransportURLReady condition
2026-03-14 09:56:26.611984 | controller |       is True\"\n        \n        # Count TransportURL CRs - should be exactly 2 (one
2026-03-14 09:56:26.611988 | controller |       for messaging, one for notifications)\n        transport_count=$(kubectl get transporturl
2026-03-14 09:56:26.611992 | controller |       -n $NAMESPACE -o name | grep \"watcher-kuttl-watcher-transport\" | wc -l)\n        notification_transport_count=$(kubectl
2026-03-14 09:56:26.611997 | controller |       get transporturl -n $NAMESPACE -o name | grep \"watcher-kuttl-watcher-notification\"
2026-03-14 09:56:26.612001 | controller |       | wc -l)\n        \n        if [ \"$transport_count\" -ne \"1\" ]; then\n          echo
2026-03-14 09:56:26.612005 | controller |       \"Expected 1 watcher-transport TransportURL, found $transport_count\"\n          exit
2026-03-14 09:56:26.612009 | controller |       1\n        fi\n        \n        if [ \"$notification_transport_count\" -ne \"1\"
2026-03-14 09:56:26.612014 | controller |       ]; then\n          echo \"Expected 1 notification-transport TransportURL, found
2026-03-14 09:56:26.612018 | controller |       $notification_transport_count\"\n          exit 1\n        fi\n        \n        echo
2026-03-14 09:56:26.612022 | controller |       \"Correctly found 2 TransportURLs (separate clusters: transport and notification)\"\n
2026-03-14 09:56:26.612026 | controller |       \       \n        # Verify watcher-transport has correct user and vhost\n        transport_user=$(kubectl
2026-03-14 09:56:26.612031 | controller |       get transporturl watcher-kuttl-watcher-transport -n $NAMESPACE -o jsonpath='{.spec.username}')\n
2026-03-14 09:56:26.612035 | controller |       \       transport_vhost=$(kubectl get transporturl watcher-kuttl-watcher-transport
2026-03-14 09:56:26.612039 | controller |       -n $NAMESPACE -o jsonpath='{.spec.vhost}')\n        if [ \"$transport_user\" !=
2026-03-14 09:56:26.612045 | controller |       \"watcher-rpc\" ]; then\n          echo \"Expected watcher-transport username 'watcher-rpc',
2026-03-14 09:56:26.612049 | controller |       found '$transport_user'\"\n          exit 1\n        fi\n        if [ \"$transport_vhost\"
2026-03-14 09:56:26.612055 | controller |       != \"watcher-rpc\" ]; then\n          echo \"Expected watcher-transport vhost 'watcher-rpc',
2026-03-14 09:56:26.612060 | controller |       found '$transport_vhost'\"\n          exit 1\n        fi\n        echo \"Watcher
2026-03-14 09:56:26.612064 | controller |       transport has correct user (watcher-rpc) and vhost (watcher-rpc)\"\n        \n        #
2026-03-14 09:56:26.612068 | controller |       Verify notification-transport has correct user and vhost\n        notif_user=$(kubectl
2026-03-14 09:56:26.612072 | controller |       get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE
2026-03-14 09:56:26.612077 | controller |       -o jsonpath='{.spec.username}')\n        notif_vhost=$(kubectl get transporturl
2026-03-14 09:56:26.612086 | controller |       watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE -o jsonpath='{.spec.vhost}')\n
2026-03-14 09:56:26.612091 | controller |       \       if [ \"$notif_user\" != \"watcher-notifications\" ]; then\n          echo
2026-03-14 09:56:26.612095 | controller |       \"Expected notification-transport username 'watcher-notifications', found '$notif_user'\"\n
2026-03-14 09:56:26.612099 | controller |       \         exit 1\n        fi\n        if [ \"$notif_vhost\" != \"watcher-notifications\"
2026-03-14 09:56:26.612103 | controller |       ]; then\n          echo \"Expected notification-transport vhost 'watcher-notifications',
2026-03-14 09:56:26.612109 | controller |       found '$notif_vhost'\"\n          exit 1\n        fi\n        echo \"Notification
2026-03-14 09:56:26.612114 | controller |       transport has correct user (watcher-notifications) and vhost (watcher-notifications)\"\n
2026-03-14 09:56:26.612118 | controller |       \       \n        # Verify that watcher.conf contains the notifications transport_url\n
2026-03-14 09:56:26.612122 | controller |       \       WATCHER_API_POD=$(kubectl get pods -n $NAMESPACE -l \"service=watcher-api\"
2026-03-14 09:56:26.612126 | controller |       -o custom-columns=:metadata.name --no-headers | grep -v ^$ | head -1)\n        if
2026-03-14 09:56:26.612131 | controller |       [ -z \"${WATCHER_API_POD}\" ]; then\n          echo \"No watcher-api pod found\"\n
2026-03-14 09:56:26.612135 | controller |       \         exit 1\n        fi\n        # Verify RPC transport_url in DEFAULT section\n
2026-03-14 09:56:26.612139 | controller |       \       rpc_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api
2026-03-14 09:56:26.612143 | controller |       -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -E '^\\[DEFAULT\\]' -A
2026-03-14 09:56:26.612148 | controller |       50 | grep 'transport_url' | head -1 || true)\n        if [ -z \"$rpc_transport_url\"
2026-03-14 09:56:26.612152 | controller |       ]; then\n          echo \"transport_url not found in DEFAULT section\"\n          exit
2026-03-14 09:56:26.612156 | controller |       1\n        fi\n        echo \"Found RPC transport_url: $rpc_transport_url\"\n        \n
2026-03-14 09:56:26.612160 | controller |       \       # Verify the RPC transport_url contains the correct vhost (watcher-rpc)\n
2026-03-14 09:56:26.612164 | controller |       \       if ! echo \"$rpc_transport_url\" | grep -q '/watcher-rpc'; then\n          echo
2026-03-14 09:56:26.612169 | controller |       \"RPC transport_url does not contain expected vhost '/watcher-rpc'\"\n          exit
2026-03-14 09:56:26.612173 | controller |       1\n        fi\n        echo \"Successfully verified vhost 'watcher-rpc' in RPC transport_url\"\n
2026-03-14 09:56:26.612177 | controller |       \       \n        # Verify the RPC transport_url contains the correct username (watcher-rpc)\n
2026-03-14 09:56:26.612182 | controller |       \       if ! echo \"$rpc_transport_url\" | grep -q 'watcher-rpc:'; then\n          echo
2026-03-14 09:56:26.612186 | controller |       \"RPC transport_url does not contain expected username 'watcher-rpc:'\"\n          exit
2026-03-14 09:56:26.612190 | controller |       1\n        fi\n        echo \"Successfully verified username 'watcher-rpc' in RPC
2026-03-14 09:56:26.612197 | controller |       transport_url\"\n        \n        # Verify oslo_messaging_notifications section
2026-03-14 09:56:26.612201 | controller |       has transport_url configured\n        notif_transport_url=$(kubectl exec -n $NAMESPACE
2026-03-14 09:56:26.612206 | controller |       ${WATCHER_API_POD} -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf
2026-03-14 09:56:26.612210 | controller |       | grep -A 5 '\\[oslo_messaging_notifications\\]' | grep 'transport_url' || true)\n
2026-03-14 09:56:26.612214 | controller |       \       if [ -z \"$notif_transport_url\" ]; then\n          echo \"transport_url
2026-03-14 09:56:26.612218 | controller |       not found in oslo_messaging_notifications section\"\n          exit 1\n        fi\n
2026-03-14 09:56:26.612223 | controller |       \       echo \"Found notifications transport_url: $notif_transport_url\"\n        \n
2026-03-14 09:56:26.612227 | controller |       \       # Verify the notifications transport_url contains the correct vhost (watcher-notifications)\n
2026-03-14 09:56:26.612231 | controller |       \       if ! echo \"$notif_transport_url\" | grep -q '/watcher-notifications'; then\n
2026-03-14 09:56:26.612236 | controller |       \         echo \"Notifications transport_url does not contain expected vhost '/watcher-notifications'\"\n
2026-03-14 09:56:26.612240 | controller |       \         exit 1\n        fi\n        echo \"Successfully verified vhost 'watcher-notifications'
2026-03-14 09:56:26.612244 | controller |       in notifications transport_url\"\n        \n        # Verify the notifications transport_url
2026-03-14 09:56:26.612249 | controller |       contains the correct username (watcher-notifications)\n        if ! echo \"$notif_transport_url\"
2026-03-14 09:56:26.612253 | controller |       | grep -q 'watcher-notifications:'; then\n          echo \"Notifications transport_url
2026-03-14 09:56:26.612257 | controller |       does not contain expected username 'watcher-notifications:'\"\n          exit 1\n
2026-03-14 09:56:26.612261 | controller |       \       fi\n        echo \"Successfully verified username 'watcher-notifications'
2026-03-14 09:56:26.612266 | controller |       in notifications transport_url\"\n        \n        exit 0\n        ]\n    logger.go:42:
2026-03-14 09:56:26.612270 | controller |       09:51:28 | watcher-rmquser/1-deploy | + kubectl wait --for=condition=Ready watcher/watcher-kuttl
2026-03-14 09:56:26.612274 | controller |       -n watcher-kuttl-default --timeout=300s\n    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612283 | controller |       | watcher.watcher.openstack.org/watcher-kuttl condition met\n    logger.go:42: 09:51:53
2026-03-14 09:56:26.612288 | controller |       | watcher-rmquser/1-deploy | + kubectl get watcher watcher-kuttl -n watcher-kuttl-default
2026-03-14 09:56:26.612292 | controller |       -o 'jsonpath={.status.conditions[?(@.type==\"WatcherNotificationTransportURLReady\")].status}'\n
2026-03-14 09:56:26.612296 | controller |       \   logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | + grep -q True\n    logger.go:42:
2026-03-14 09:56:26.612302 | controller |       09:51:53 | watcher-rmquser/1-deploy | + echo 'WatcherNotificationTransportURLReady
2026-03-14 09:56:26.612307 | controller |       condition is True'\n    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | WatcherNotificationTransportURLReady
2026-03-14 09:56:26.612311 | controller |       condition is True\n    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | ++ wc
2026-03-14 09:56:26.612316 | controller |       -l\n    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-transport\n
2026-03-14 09:56:26.612320 | controller |       \   logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:56:26.612324 | controller |       -n watcher-kuttl-default -o name\n    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612328 | controller |       | + transport_count=1\n    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | ++
2026-03-14 09:56:26.612333 | controller |       wc -l\n    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-notification\n
2026-03-14 09:56:26.612337 | controller |       \   logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:56:26.612341 | controller |       -n watcher-kuttl-default -o name\n    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612347 | controller |       | + notification_transport_count=1\n    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612351 | controller |       | + '[' 1 -ne 1 ']'\n    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | + '['
2026-03-14 09:56:26.612356 | controller |       1 -ne 1 ']'\n    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | + echo 'Correctly
2026-03-14 09:56:26.612360 | controller |       found 2 TransportURLs (separate clusters: transport and notification)'\n    logger.go:42:
2026-03-14 09:56:26.612364 | controller |       09:51:53 | watcher-rmquser/1-deploy | Correctly found 2 TransportURLs (separate
2026-03-14 09:56:26.612368 | controller |       clusters: transport and notification)\n    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612373 | controller |       | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default
2026-03-14 09:56:26.612377 | controller |       -o 'jsonpath={.spec.username}'\n    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612381 | controller |       | + transport_user=watcher-rpc\n    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612385 | controller |       | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default
2026-03-14 09:56:26.612390 | controller |       -o 'jsonpath={.spec.vhost}'\n    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612394 | controller |       | + transport_vhost=watcher-rpc\n    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612398 | controller |       | + '[' watcher-rpc '!=' watcher-rpc ']'\n    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612402 | controller |       | + '[' watcher-rpc '!=' watcher-rpc ']'\n    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612407 | controller |       | + echo 'Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)'\n
2026-03-14 09:56:26.612411 | controller |       \   logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | Watcher transport has correct
2026-03-14 09:56:26.612415 | controller |       user (watcher-rpc) and vhost (watcher-rpc)\n    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612420 | controller |       | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications
2026-03-14 09:56:26.612424 | controller |       -n watcher-kuttl-default -o 'jsonpath={.spec.username}'\n    logger.go:42: 09:51:54
2026-03-14 09:56:26.612428 | controller |       | watcher-rmquser/1-deploy | + notif_user=watcher-notifications\n    logger.go:42:
2026-03-14 09:56:26.612432 | controller |       09:51:54 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications
2026-03-14 09:56:26.612437 | controller |       -n watcher-kuttl-default -o 'jsonpath={.spec.vhost}'\n    logger.go:42: 09:51:54
2026-03-14 09:56:26.612441 | controller |       | watcher-rmquser/1-deploy | + notif_vhost=watcher-notifications\n    logger.go:42:
2026-03-14 09:56:26.612445 | controller |       09:51:54 | watcher-rmquser/1-deploy | + '[' watcher-notifications '!=' watcher-notifications
2026-03-14 09:56:26.612451 | controller |       ']'\n    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + '[' watcher-notifications
2026-03-14 09:56:26.612458 | controller |       '!=' watcher-notifications ']'\n    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612464 | controller |       | + echo 'Notification transport has correct user (watcher-notifications) and vhost
2026-03-14 09:56:26.612469 | controller |       (watcher-notifications)'\n    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612476 | controller |       | Notification transport has correct user (watcher-notifications) and vhost (watcher-notifications)\n
2026-03-14 09:56:26.612488 | controller |       \   logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | ++ grep -v '^$'\n    logger.go:42:
2026-03-14 09:56:26.612492 | controller |       09:51:54 | watcher-rmquser/1-deploy | ++ kubectl get pods -n watcher-kuttl-default
2026-03-14 09:56:26.612497 | controller |       -l service=watcher-api -o custom-columns=:metadata.name --no-headers\n    logger.go:42:
2026-03-14 09:56:26.612501 | controller |       09:51:54 | watcher-rmquser/1-deploy | ++ head -1\n    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612508 | controller |       | + WATCHER_API_POD=watcher-kuttl-api-0\n    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612512 | controller |       | + '[' -z watcher-kuttl-api-0 ']'\n    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612516 | controller |       | ++ head -1\n    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | ++ grep transport_url\n
2026-03-14 09:56:26.612520 | controller |       \   logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | ++ grep -E '^\\[DEFAULT\\]'
2026-03-14 09:56:26.612525 | controller |       -A 50\n    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | ++ kubectl exec -n
2026-03-14 09:56:26.612529 | controller |       watcher-kuttl-default watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.612535 | controller |       \   logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + rpc_transport_url='transport_url
2026-03-14 09:56:26.612540 | controller |       = rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n
2026-03-14 09:56:26.612545 | controller |       \   logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + '[' -z 'transport_url
2026-03-14 09:56:26.612550 | controller |       = rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'
2026-03-14 09:56:26.612554 | controller |       ']'\n    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + echo 'Found RPC transport_url:
2026-03-14 09:56:26.612558 | controller |       transport_url = rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n
2026-03-14 09:56:26.612562 | controller |       \   logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | Found RPC transport_url:
2026-03-14 09:56:26.612567 | controller |       transport_url = rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1\n
2026-03-14 09:56:26.612571 | controller |       \   logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + grep -q /watcher-rpc\n
2026-03-14 09:56:26.612575 | controller |       \   logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + echo 'transport_url =
2026-03-14 09:56:26.612580 | controller |       rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n
2026-03-14 09:56:26.612584 | controller |       \   logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + echo 'Successfully verified
2026-03-14 09:56:26.612588 | controller |       vhost '\\''watcher-rpc'\\'' in RPC transport_url'\n    logger.go:42: 09:51:54 |
2026-03-14 09:56:26.612593 | controller |       watcher-rmquser/1-deploy | Successfully verified vhost 'watcher-rpc' in RPC transport_url\n
2026-03-14 09:56:26.612610 | controller |       \   logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + grep -q watcher-rpc:\n
2026-03-14 09:56:26.612614 | controller |       \   logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + echo 'transport_url =
2026-03-14 09:56:26.612619 | controller |       rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n
2026-03-14 09:56:26.612623 | controller |       \   logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + echo 'Successfully verified
2026-03-14 09:56:26.612627 | controller |       username '\\''watcher-rpc'\\'' in RPC transport_url'\n    logger.go:42: 09:51:54
2026-03-14 09:56:26.612632 | controller |       | watcher-rmquser/1-deploy | Successfully verified username 'watcher-rpc' in RPC
2026-03-14 09:56:26.612636 | controller |       transport_url\n    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | ++ grep -A
2026-03-14 09:56:26.612640 | controller |       5 '\\[oslo_messaging_notifications\\]'\n    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612644 | controller |       | ++ grep transport_url\n    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy |
2026-03-14 09:56:26.612649 | controller |       ++ kubectl exec -n watcher-kuttl-default watcher-kuttl-api-0 -c watcher-api -- cat
2026-03-14 09:56:26.612653 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42: 09:51:55 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612659 | controller |       | + notif_transport_url='transport_url = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n
2026-03-14 09:56:26.612668 | controller |       \   logger.go:42: 09:51:55 | watcher-rmquser/1-deploy | + '[' -z 'transport_url
2026-03-14 09:56:26.612674 | controller |       = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'
2026-03-14 09:56:26.612679 | controller |       ']'\n    logger.go:42: 09:51:55 | watcher-rmquser/1-deploy | + echo 'Found notifications
2026-03-14 09:56:26.612691 | controller |       transport_url: transport_url = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n
2026-03-14 09:56:26.612697 | controller |       \   logger.go:42: 09:51:55 | watcher-rmquser/1-deploy | Found notifications transport_url:
2026-03-14 09:56:26.612704 | controller |       transport_url = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1\n
2026-03-14 09:56:26.612709 | controller |       \   logger.go:42: 09:51:55 | watcher-rmquser/1-deploy | + grep -q /watcher-notifications\n
2026-03-14 09:56:26.612713 | controller |       \   logger.go:42: 09:51:55 | watcher-rmquser/1-deploy | + echo 'transport_url =
2026-03-14 09:56:26.612717 | controller |       rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n
2026-03-14 09:56:26.612722 | controller |       \   logger.go:42: 09:51:55 | watcher-rmquser/1-deploy | + echo 'Successfully verified
2026-03-14 09:56:26.612729 | controller |       vhost '\\''watcher-notifications'\\'' in notifications transport_url'\n    logger.go:42:
2026-03-14 09:56:26.612733 | controller |       09:51:55 | watcher-rmquser/1-deploy | Successfully verified vhost 'watcher-notifications'
2026-03-14 09:56:26.612737 | controller |       in notifications transport_url\n    logger.go:42: 09:51:55 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612742 | controller |       | + grep -q watcher-notifications:\n    logger.go:42: 09:51:55 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612746 | controller |       | + echo 'transport_url = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n
2026-03-14 09:56:26.612750 | controller |       \   logger.go:42: 09:51:55 | watcher-rmquser/1-deploy | + echo 'Successfully verified
2026-03-14 09:56:26.612754 | controller |       username '\\''watcher-notifications'\\'' in notifications transport_url'\n    logger.go:42:
2026-03-14 09:56:26.612759 | controller |       09:51:55 | watcher-rmquser/1-deploy | Successfully verified username 'watcher-notifications'
2026-03-14 09:56:26.612763 | controller |       in notifications transport_url\n    logger.go:42: 09:51:55 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.612767 | controller |       | + exit 0\n    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | running command:
2026-03-14 09:56:26.612771 | controller |       [sh -c set -euxo pipefail\n        \n        # Wait for Watcher to be Ready\n        kubectl
2026-03-14 09:56:26.612776 | controller |       wait --for=condition=Ready watcher/watcher-kuttl -n $NAMESPACE --timeout=300s\n
2026-03-14 09:56:26.612780 | controller |       \       \n        # Verify WatcherNotificationTransportURLReady condition exists
2026-03-14 09:56:26.612786 | controller |       and is True\n        kubectl get watcher watcher-kuttl -n $NAMESPACE -o jsonpath='{.status.conditions[?(@.type==\"WatcherNotificationTransportURLReady\")].status}'
2026-03-14 09:56:26.612790 | controller |       | grep -q \"True\"\n        echo \"WatcherNotificationTransportURLReady condition
2026-03-14 09:56:26.612795 | controller |       is True\"\n        \n        # Count TransportURL CRs - should be exactly 2 (one
2026-03-14 09:56:26.612799 | controller |       for messaging, one for notifications)\n        transport_count=$(kubectl get transporturl
2026-03-14 09:56:26.612803 | controller |       -n $NAMESPACE -o name | grep \"watcher-kuttl-watcher-transport\" | wc -l)\n        notification_transport_count=$(kubectl
2026-03-14 09:56:26.612807 | controller |       get transporturl -n $NAMESPACE -o name | grep \"watcher-kuttl-watcher-notification\"
2026-03-14 09:56:26.612813 | controller |       | wc -l)\n        \n        if [ \"$transport_count\" -ne \"1\" ]; then\n          echo
2026-03-14 09:56:26.612818 | controller |       \"Expected 1 watcher-transport TransportURL, found $transport_count\"\n          exit
2026-03-14 09:56:26.612822 | controller |       1\n        fi\n        \n        if [ \"$notification_transport_count\" -ne \"1\"
2026-03-14 09:56:26.612828 | controller |       ]; then\n          echo \"Expected 1 notification-transport TransportURL, found
2026-03-14 09:56:26.612832 | controller |       $notification_transport_count\"\n          exit 1\n        fi\n        \n        echo
2026-03-14 09:56:26.612836 | controller |       \"Correctly found 2 TransportURLs (separate clusters: transport and notification)\"\n
2026-03-14 09:56:26.612841 | controller |       \       \n        # Verify watcher-transport has correct user and vhost\n        transport_user=$(kubectl
2026-03-14 09:56:26.612845 | controller |       get transporturl watcher-kuttl-watcher-transport -n $NAMESPACE -o jsonpath='{.spec.username}')\n
2026-03-14 09:56:26.612849 | controller |       \       transport_vhost=$(kubectl get transporturl watcher-kuttl-watcher-transport
2026-03-14 09:56:26.612853 | controller |       -n $NAMESPACE -o jsonpath='{.spec.vhost}')\n        if [ \"$transport_user\" !=
2026-03-14 09:56:26.612858 | controller |       \"watcher-rpc\" ]; then\n          echo \"Expected watcher-transport username 'watcher-rpc',
2026-03-14 09:56:26.612862 | controller |       found '$transport_user'\"\n          exit 1\n        fi\n        if [ \"$transport_vhost\"
2026-03-14 09:56:26.612866 | controller |       != \"watcher-rpc\" ]; then\n          echo \"Expected watcher-transport vhost 'watcher-rpc',
2026-03-14 09:56:26.612871 | controller |       found '$transport_vhost'\"\n          exit 1\n        fi\n        echo \"Watcher
2026-03-14 09:56:26.612881 | controller |       transport has correct user (watcher-rpc) and vhost (watcher-rpc)\"\n        \n        #
2026-03-14 09:56:26.612886 | controller |       Verify notification-transport has correct user and vhost\n        notif_user=$(kubectl
2026-03-14 09:56:26.612890 | controller |       get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE
2026-03-14 09:56:26.612894 | controller |       -o jsonpath='{.spec.username}')\n        notif_vhost=$(kubectl get transporturl
2026-03-14 09:56:26.612898 | controller |       watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE -o jsonpath='{.spec.vhost}')\n
2026-03-14 09:56:26.612903 | controller |       \       if [ \"$notif_user\" != \"watcher-notifications\" ]; then\n          echo
2026-03-14 09:56:26.612907 | controller |       \"Expected notification-transport username 'watcher-notifications', found '$notif_user'\"\n
2026-03-14 09:56:26.612911 | controller |       \         exit 1\n        fi\n        if [ \"$notif_vhost\" != \"watcher-notifications\"
2026-03-14 09:56:26.612916 | controller |       ]; then\n          echo \"Expected notification-transport vhost 'watcher-notifications',
2026-03-14 09:56:26.612920 | controller |       found '$notif_vhost'\"\n          exit 1\n        fi\n        echo \"Notification
2026-03-14 09:56:26.612924 | controller |       transport has correct user (watcher-notifications) and vhost (watcher-notifications)\"\n
2026-03-14 09:56:26.612928 | controller |       \       \n        # Verify that watcher.conf contains the notifications transport_url\n
2026-03-14 09:56:26.612933 | controller |       \       WATCHER_API_POD=$(kubectl get pods -n $NAMESPACE -l \"service=watcher-api\"
2026-03-14 09:56:26.612937 | controller |       -o custom-columns=:metadata.name --no-headers | grep -v ^$ | head -1)\n        if
2026-03-14 09:56:26.612941 | controller |       [ -z \"${WATCHER_API_POD}\" ]; then\n          echo \"No watcher-api pod found\"\n
2026-03-14 09:56:26.612946 | controller |       \         exit 1\n        fi\n        # Verify RPC transport_url in DEFAULT section\n
2026-03-14 09:56:26.612950 | controller |       \       rpc_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api
2026-03-14 09:56:26.612954 | controller |       -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -E '^\\[DEFAULT\\]' -A
2026-03-14 09:56:26.612959 | controller |       50 | grep 'transport_url' | head -1 || true)\n        if [ -z \"$rpc_transport_url\"
2026-03-14 09:56:26.612965 | controller |       ]; then\n          echo \"transport_url not found in DEFAULT section\"\n          exit
2026-03-14 09:56:26.612969 | controller |       1\n        fi\n        echo \"Found RPC transport_url: $rpc_transport_url\"\n        \n
2026-03-14 09:56:26.612973 | controller |       \       # Verify the RPC transport_url contains the correct vhost (watcher-rpc)\n
2026-03-14 09:56:26.612977 | controller |       \       if ! echo \"$rpc_transport_url\" | grep -q '/watcher-rpc'; then\n          echo
2026-03-14 09:56:26.612982 | controller |       \"RPC transport_url does not contain expected vhost '/watcher-rpc'\"\n          exit
2026-03-14 09:56:26.612986 | controller |       1\n        fi\n        echo \"Successfully verified vhost 'watcher-rpc' in RPC transport_url\"\n
2026-03-14 09:56:26.612990 | controller |       \       \n        # Verify the RPC transport_url contains the correct username (watcher-rpc)\n
2026-03-14 09:56:26.612994 | controller |       \       if ! echo \"$rpc_transport_url\" | grep -q 'watcher-rpc:'; then\n          echo
2026-03-14 09:56:26.612999 | controller |       \"RPC transport_url does not contain expected username 'watcher-rpc:'\"\n          exit
2026-03-14 09:56:26.613003 | controller |       1\n        fi\n        echo \"Successfully verified username 'watcher-rpc' in RPC
2026-03-14 09:56:26.613009 | controller |       transport_url\"\n        \n        # Verify oslo_messaging_notifications section
2026-03-14 09:56:26.613014 | controller |       has transport_url configured\n        notif_transport_url=$(kubectl exec -n $NAMESPACE
2026-03-14 09:56:26.613018 | controller |       ${WATCHER_API_POD} -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf
2026-03-14 09:56:26.613022 | controller |       | grep -A 5 '\\[oslo_messaging_notifications\\]' | grep 'transport_url' || true)\n
2026-03-14 09:56:26.613027 | controller |       \       if [ -z \"$notif_transport_url\" ]; then\n          echo \"transport_url
2026-03-14 09:56:26.613031 | controller |       not found in oslo_messaging_notifications section\"\n          exit 1\n        fi\n
2026-03-14 09:56:26.613035 | controller |       \       echo \"Found notifications transport_url: $notif_transport_url\"\n        \n
2026-03-14 09:56:26.613040 | controller |       \       # Verify the notifications transport_url contains the correct vhost (watcher-notifications)\n
2026-03-14 09:56:26.613044 | controller |       \       if ! echo \"$notif_transport_url\" | grep -q '/watcher-notifications'; then\n
2026-03-14 09:56:26.613048 | controller |       \         echo \"Notifications transport_url does not contain expected vhost '/watcher-notifications'\"\n
2026-03-14 09:56:26.613052 | controller |       \         exit 1\n        fi\n        echo \"Successfully verified vhost 'watcher-notifications'
2026-03-14 09:56:26.613057 | controller |       in notifications transport_url\"\n        \n        # Verify the notifications transport_url
2026-03-14 09:56:26.613061 | controller |       contains the correct username (watcher-notifications)\n        if ! echo \"$notif_transport_url\"
2026-03-14 09:56:26.613065 | controller |       | grep -q 'watcher-notifications:'; then\n          echo \"Notifications transport_url
2026-03-14 09:56:26.613070 | controller |       does not contain expected username 'watcher-notifications:'\"\n          exit 1\n
2026-03-14 09:56:26.613079 | controller |       \       fi\n        echo \"Successfully verified username 'watcher-notifications'
2026-03-14 09:56:26.613083 | controller |       in notifications transport_url\"\n        \n        exit 0\n        ]\n    logger.go:42:
2026-03-14 09:56:26.613088 | controller |       09:51:56 | watcher-rmquser/1-deploy | + kubectl wait --for=condition=Ready watcher/watcher-kuttl
2026-03-14 09:56:26.613092 | controller |       -n watcher-kuttl-default --timeout=300s\n    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613096 | controller |       | watcher.watcher.openstack.org/watcher-kuttl condition met\n    logger.go:42: 09:51:56
2026-03-14 09:56:26.613101 | controller |       | watcher-rmquser/1-deploy | + kubectl get watcher watcher-kuttl -n watcher-kuttl-default
2026-03-14 09:56:26.613107 | controller |       -o 'jsonpath={.status.conditions[?(@.type==\"WatcherNotificationTransportURLReady\")].status}'\n
2026-03-14 09:56:26.613111 | controller |       \   logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | + grep -q True\n    logger.go:42:
2026-03-14 09:56:26.613117 | controller |       09:51:56 | watcher-rmquser/1-deploy | + echo 'WatcherNotificationTransportURLReady
2026-03-14 09:56:26.613121 | controller |       condition is True'\n    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | WatcherNotificationTransportURLReady
2026-03-14 09:56:26.613126 | controller |       condition is True\n    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | ++ kubectl
2026-03-14 09:56:26.613130 | controller |       get transporturl -n watcher-kuttl-default -o name\n    logger.go:42: 09:51:56 |
2026-03-14 09:56:26.613134 | controller |       watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-transport\n    logger.go:42:
2026-03-14 09:56:26.613138 | controller |       09:51:56 | watcher-rmquser/1-deploy | ++ wc -l\n    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613143 | controller |       | + transport_count=1\n    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | ++
2026-03-14 09:56:26.613147 | controller |       grep watcher-kuttl-watcher-notification\n    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613151 | controller |       | ++ wc -l\n    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | ++ kubectl get
2026-03-14 09:56:26.613155 | controller |       transporturl -n watcher-kuttl-default -o name\n    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613160 | controller |       | + notification_transport_count=1\n    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613164 | controller |       | + '[' 1 -ne 1 ']'\n    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | + '['
2026-03-14 09:56:26.613168 | controller |       1 -ne 1 ']'\n    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | + echo 'Correctly
2026-03-14 09:56:26.613172 | controller |       found 2 TransportURLs (separate clusters: transport and notification)'\n    logger.go:42:
2026-03-14 09:56:26.613177 | controller |       09:51:56 | watcher-rmquser/1-deploy | Correctly found 2 TransportURLs (separate
2026-03-14 09:56:26.613181 | controller |       clusters: transport and notification)\n    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613185 | controller |       | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default
2026-03-14 09:56:26.613189 | controller |       -o 'jsonpath={.spec.username}'\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613194 | controller |       | + transport_user=watcher-rpc\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613198 | controller |       | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default
2026-03-14 09:56:26.613202 | controller |       -o 'jsonpath={.spec.vhost}'\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613206 | controller |       | + transport_vhost=watcher-rpc\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613211 | controller |       | + '[' watcher-rpc '!=' watcher-rpc ']'\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613215 | controller |       | + '[' watcher-rpc '!=' watcher-rpc ']'\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613219 | controller |       | + echo 'Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)'\n
2026-03-14 09:56:26.613223 | controller |       \   logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | Watcher transport has correct
2026-03-14 09:56:26.613228 | controller |       user (watcher-rpc) and vhost (watcher-rpc)\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613232 | controller |       | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications
2026-03-14 09:56:26.613239 | controller |       -n watcher-kuttl-default -o 'jsonpath={.spec.username}'\n    logger.go:42: 09:51:57
2026-03-14 09:56:26.613244 | controller |       | watcher-rmquser/1-deploy | + notif_user=watcher-notifications\n    logger.go:42:
2026-03-14 09:56:26.613248 | controller |       09:51:57 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications
2026-03-14 09:56:26.613252 | controller |       -n watcher-kuttl-default -o 'jsonpath={.spec.vhost}'\n    logger.go:42: 09:51:57
2026-03-14 09:56:26.613257 | controller |       | watcher-rmquser/1-deploy | + notif_vhost=watcher-notifications\n    logger.go:42:
2026-03-14 09:56:26.613262 | controller |       09:51:57 | watcher-rmquser/1-deploy | + '[' watcher-notifications '!=' watcher-notifications
2026-03-14 09:56:26.613272 | controller |       ']'\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + '[' watcher-notifications
2026-03-14 09:56:26.613276 | controller |       '!=' watcher-notifications ']'\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613281 | controller |       | + echo 'Notification transport has correct user (watcher-notifications) and vhost
2026-03-14 09:56:26.613285 | controller |       (watcher-notifications)'\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613289 | controller |       | Notification transport has correct user (watcher-notifications) and vhost (watcher-notifications)\n
2026-03-14 09:56:26.613294 | controller |       \   logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | ++ grep -v '^$'\n    logger.go:42:
2026-03-14 09:56:26.613298 | controller |       09:51:57 | watcher-rmquser/1-deploy | ++ head -1\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613302 | controller |       | ++ kubectl get pods -n watcher-kuttl-default -l service=watcher-api -o custom-columns=:metadata.name
2026-03-14 09:56:26.613306 | controller |       --no-headers\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + WATCHER_API_POD=watcher-kuttl-api-0\n
2026-03-14 09:56:26.613311 | controller |       \   logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + '[' -z watcher-kuttl-api-0
2026-03-14 09:56:26.613315 | controller |       ']'\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | ++ grep -E '^\\[DEFAULT\\]'
2026-03-14 09:56:26.613319 | controller |       -A 50\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | ++ grep transport_url\n
2026-03-14 09:56:26.613324 | controller |       \   logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default
2026-03-14 09:56:26.613328 | controller |       watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.613332 | controller |       \   logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | ++ head -1\n    logger.go:42:
2026-03-14 09:56:26.613336 | controller |       09:51:57 | watcher-rmquser/1-deploy | + rpc_transport_url='transport_url = rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n
2026-03-14 09:56:26.613341 | controller |       \   logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + '[' -z 'transport_url
2026-03-14 09:56:26.613345 | controller |       = rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'
2026-03-14 09:56:26.613349 | controller |       ']'\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo 'Found RPC transport_url:
2026-03-14 09:56:26.613354 | controller |       transport_url = rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n
2026-03-14 09:56:26.613358 | controller |       \   logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | Found RPC transport_url:
2026-03-14 09:56:26.613362 | controller |       transport_url = rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1\n
2026-03-14 09:56:26.613367 | controller |       \   logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo 'transport_url =
2026-03-14 09:56:26.613371 | controller |       rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n
2026-03-14 09:56:26.613375 | controller |       \   logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + grep -q /watcher-rpc\n
2026-03-14 09:56:26.613379 | controller |       \   logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo 'Successfully verified
2026-03-14 09:56:26.613384 | controller |       vhost '\\''watcher-rpc'\\'' in RPC transport_url'\n    logger.go:42: 09:51:57 |
2026-03-14 09:56:26.613388 | controller |       watcher-rmquser/1-deploy | Successfully verified vhost 'watcher-rpc' in RPC transport_url\n
2026-03-14 09:56:26.613392 | controller |       \   logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + grep -q watcher-rpc:\n
2026-03-14 09:56:26.613396 | controller |       \   logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo 'transport_url =
2026-03-14 09:56:26.613403 | controller |       rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n
2026-03-14 09:56:26.613407 | controller |       \   logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo 'Successfully verified
2026-03-14 09:56:26.613411 | controller |       username '\\''watcher-rpc'\\'' in RPC transport_url'\n    logger.go:42: 09:51:57
2026-03-14 09:56:26.613416 | controller |       | watcher-rmquser/1-deploy | Successfully verified username 'watcher-rpc' in RPC
2026-03-14 09:56:26.613420 | controller |       transport_url\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | ++ grep -A
2026-03-14 09:56:26.613424 | controller |       5 '\\[oslo_messaging_notifications\\]'\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613428 | controller |       | ++ kubectl exec -n watcher-kuttl-default watcher-kuttl-api-0 -c watcher-api --
2026-03-14 09:56:26.613433 | controller |       cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613437 | controller |       | ++ grep transport_url\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy |
2026-03-14 09:56:26.613441 | controller |       + notif_transport_url='transport_url = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n
2026-03-14 09:56:26.613450 | controller |       \   logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + '[' -z 'transport_url
2026-03-14 09:56:26.613455 | controller |       = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'
2026-03-14 09:56:26.613459 | controller |       ']'\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo 'Found notifications
2026-03-14 09:56:26.613464 | controller |       transport_url: transport_url = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n
2026-03-14 09:56:26.613468 | controller |       \   logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | Found notifications transport_url:
2026-03-14 09:56:26.613472 | controller |       transport_url = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1\n
2026-03-14 09:56:26.613477 | controller |       \   logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + grep -q /watcher-notifications\n
2026-03-14 09:56:26.613481 | controller |       \   logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo 'transport_url =
2026-03-14 09:56:26.613485 | controller |       rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n
2026-03-14 09:56:26.613490 | controller |       \   logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo 'Successfully verified
2026-03-14 09:56:26.613496 | controller |       vhost '\\''watcher-notifications'\\'' in notifications transport_url'\n    logger.go:42:
2026-03-14 09:56:26.613500 | controller |       09:51:57 | watcher-rmquser/1-deploy | Successfully verified vhost 'watcher-notifications'
2026-03-14 09:56:26.613505 | controller |       in notifications transport_url\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613509 | controller |       | + grep -q watcher-notifications:\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613513 | controller |       | + echo 'transport_url = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n
2026-03-14 09:56:26.613518 | controller |       \   logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo 'Successfully verified
2026-03-14 09:56:26.613522 | controller |       username '\\''watcher-notifications'\\'' in notifications transport_url'\n    logger.go:42:
2026-03-14 09:56:26.613526 | controller |       09:51:57 | watcher-rmquser/1-deploy | Successfully verified username 'watcher-notifications'
2026-03-14 09:56:26.613530 | controller |       in notifications transport_url\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy
2026-03-14 09:56:26.613536 | controller |       | + exit 0\n    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | test step completed
2026-03-14 09:56:26.613541 | controller |       1-deploy\n    logger.go:42: 09:51:57 | watcher-rmquser/2-cleanup-watcher | starting
2026-03-14 09:56:26.613545 | controller |       test step 2-cleanup-watcher\n    logger.go:42: 09:52:03 | watcher-rmquser/2-cleanup-watcher
2026-03-14 09:56:26.613549 | controller |       | test step completed 2-cleanup-watcher\n    logger.go:42: 09:52:03 | watcher-rmquser
2026-03-14 09:56:26.613553 | controller |       | skipping kubernetes event logging\n=== CONT  kuttl/harness/watcher\n    logger.go:42:
2026-03-14 09:56:26.613559 | controller |       09:52:03 | watcher | Skipping creation of user-supplied namespace: watcher-kuttl-default\n
2026-03-14 09:56:26.613564 | controller |       \   logger.go:42: 09:52:03 | watcher/0-cleanup-watcher | starting test step 0-cleanup-watcher\n
2026-03-14 09:56:26.613568 | controller |       \   logger.go:42: 09:52:03 | watcher/0-cleanup-watcher | test step completed 0-cleanup-watcher\n
2026-03-14 09:56:26.613572 | controller |       \   logger.go:42: 09:52:03 | watcher/1-deploy-with-defaults | starting test step
2026-03-14 09:56:26.613577 | controller |       1-deploy-with-defaults\n    logger.go:42: 09:52:03 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.613581 | controller |       | Watcher:watcher-kuttl-default/watcher-kuttl created\n    logger.go:42: 09:52:04
2026-03-14 09:56:26.613585 | controller |       | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:56:26.613589 | controller |       \       oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.613604 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc exec
2026-03-14 09:56:26.613611 | controller |       -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.613615 | controller |       -c Type -c ID | grep watcher| awk '{print $1}')\n        [ $(oc get -n watcher-kuttl-default
2026-03-14 09:56:26.613620 | controller |       keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [
2026-03-14 09:56:26.613624 | controller |       -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\"
2026-03-14 09:56:26.613628 | controller |       ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.613632 | controller |       -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n        [ \"$(oc
2026-03-14 09:56:26.613642 | controller |       get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64
2026-03-14 09:56:26.613647 | controller |       -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2
2026-03-14 09:56:26.613651 | controller |       ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.613656 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\"
2026-03-14 09:56:26.613660 | controller |       -ge 1 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:56:26.613664 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:56:26.613668 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:56:26.613673 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:56:26.613677 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:56:26.613681 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:56:26.613686 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:56:26.613690 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:56:26.613694 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:56:26.613698 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:52:04 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.613705 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.613709 | controller |       value -c Name -c Type\n    logger.go:42: 09:52:04 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.613714 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:52:06 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.613718 | controller |       | + '[' 0 == 1 ']'\n    logger.go:42: 09:52:07 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.613724 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.613729 | controller |       openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c
2026-03-14 09:56:26.613733 | controller |       ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.613737 | controller |       -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print
2026-03-14 09:56:26.613741 | controller |       $1}')\n        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.613746 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl
2026-03-14 09:56:26.613750 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        [ \"$(oc get -n watcher-kuttl-default
2026-03-14 09:56:26.613754 | controller |       secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep
2026-03-14 09:56:26.613759 | controller |       -c 'ssl=1')\" == 1 ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.613763 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\"
2026-03-14 09:56:26.613767 | controller |       == 2 ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.613771 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\"
2026-03-14 09:56:26.613776 | controller |       -ge 1 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:56:26.613780 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:56:26.613784 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:56:26.613789 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:56:26.613793 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:56:26.613797 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:56:26.613801 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:56:26.613806 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:56:26.613810 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:56:26.613814 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:52:07 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.613819 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.613823 | controller |       value -c Name -c Type\n    logger.go:42: 09:52:07 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.613827 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:52:09 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.613836 | controller |       | + '[' 0 == 1 ']'\n    logger.go:42: 09:52:10 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.613841 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.613845 | controller |       openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c
2026-03-14 09:56:26.613850 | controller |       ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.613856 | controller |       -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print
2026-03-14 09:56:26.613860 | controller |       $1}')\n        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.613864 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl
2026-03-14 09:56:26.613869 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        [ \"$(oc get -n watcher-kuttl-default
2026-03-14 09:56:26.613873 | controller |       secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep
2026-03-14 09:56:26.613877 | controller |       -c 'ssl=1')\" == 1 ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.613882 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\"
2026-03-14 09:56:26.613886 | controller |       == 2 ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.613890 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\"
2026-03-14 09:56:26.613895 | controller |       -ge 1 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:56:26.613899 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:56:26.613903 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:56:26.613907 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:56:26.613912 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:56:26.613916 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:56:26.613920 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:56:26.613925 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:56:26.613929 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:56:26.613933 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:52:10 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.613937 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.613943 | controller |       value -c Name -c Type\n    logger.go:42: 09:52:10 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.613948 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:52:13 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.613952 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:52:13 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.613956 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.613961 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:52:13 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.613965 | controller |       | ++ grep watcher\n    logger.go:42: 09:52:13 | watcher/1-deploy-with-defaults |
2026-03-14 09:56:26.613969 | controller |       ++ awk '{print $1}'\n    logger.go:42: 09:52:15 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.613973 | controller |       | + SERVICEID=e32ab6f2cb3c4ec69d47af38a3e246ce\n    logger.go:42: 09:52:15 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.613978 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.613982 | controller |       \   logger.go:42: 09:52:15 | watcher/1-deploy-with-defaults | + '[' e32ab6f2cb3c4ec69d47af38a3e246ce
2026-03-14 09:56:26.613986 | controller |       == e32ab6f2cb3c4ec69d47af38a3e246ce ']'\n    logger.go:42: 09:52:15 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.613991 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:56:26.613997 | controller |       \   logger.go:42: 09:52:15 | watcher/1-deploy-with-defaults | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.614001 | controller |       ']'\n    logger.go:42: 09:52:15 | watcher/1-deploy-with-defaults | ++ base64 -d\n
2026-03-14 09:56:26.614007 | controller |       \   logger.go:42: 09:52:15 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.614011 | controller |       secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n    logger.go:42:
2026-03-14 09:56:26.614016 | controller |       09:52:15 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1\n    logger.go:42:
2026-03-14 09:56:26.614020 | controller |       09:52:16 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:56:26.614029 | controller |       09:52:16 | watcher/1-deploy-with-defaults | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n
2026-03-14 09:56:26.614034 | controller |       \   logger.go:42: 09:52:16 | watcher/1-deploy-with-defaults | ++ base64 -d\n    logger.go:42:
2026-03-14 09:56:26.614038 | controller |       09:52:16 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret
2026-03-14 09:56:26.614042 | controller |       watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42:
2026-03-14 09:56:26.614047 | controller |       09:52:16 | watcher/1-deploy-with-defaults | + '[' 2 == 2 ']'\n    logger.go:42:
2026-03-14 09:56:26.614051 | controller |       09:52:16 | watcher/1-deploy-with-defaults | ++ base64 -d\n    logger.go:42: 09:52:16
2026-03-14 09:56:26.614055 | controller |       | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.614060 | controller |       -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42: 09:52:16 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614064 | controller |       | ++ grep -c 'region_name = regionOne'\n    logger.go:42: 09:52:16 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614068 | controller |       | + '[' 9 -ge 1 ']'\n    logger.go:42: 09:52:16 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614072 | controller |       | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n
2026-03-14 09:56:26.614077 | controller |       \   logger.go:42: 09:52:16 | watcher/1-deploy-with-defaults | + '[' '' == '' ']'\n
2026-03-14 09:56:26.614081 | controller |       \   logger.go:42: 09:52:16 | watcher/1-deploy-with-defaults | + exit 0\n    logger.go:42:
2026-03-14 09:56:26.614085 | controller |       09:52:17 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:56:26.614089 | controller |       \       oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.614094 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc exec
2026-03-14 09:56:26.614098 | controller |       -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.614102 | controller |       -c Type -c ID | grep watcher| awk '{print $1}')\n        [ $(oc get -n watcher-kuttl-default
2026-03-14 09:56:26.614107 | controller |       keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [
2026-03-14 09:56:26.614111 | controller |       -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\"
2026-03-14 09:56:26.614115 | controller |       ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.614119 | controller |       -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n        [ \"$(oc
2026-03-14 09:56:26.614124 | controller |       get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64
2026-03-14 09:56:26.614128 | controller |       -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2
2026-03-14 09:56:26.614132 | controller |       ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.614137 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\"
2026-03-14 09:56:26.614143 | controller |       -ge 1 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:56:26.614147 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:56:26.614151 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:56:26.614156 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:56:26.614160 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:56:26.614164 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:56:26.614170 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:56:26.614175 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:56:26.614179 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:56:26.614183 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:52:17 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614188 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.614192 | controller |       value -c Name -c Type\n    logger.go:42: 09:52:17 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614196 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:52:19 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614200 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:52:19 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614205 | controller |       | ++ grep watcher\n    logger.go:42: 09:52:19 | watcher/1-deploy-with-defaults |
2026-03-14 09:56:26.614209 | controller |       ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.614213 | controller |       value -c Name -c Type -c ID\n    logger.go:42: 09:52:19 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614223 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614227 | controller |       | + SERVICEID=e32ab6f2cb3c4ec69d47af38a3e246ce\n    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614232 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.614236 | controller |       \   logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | + '[' e32ab6f2cb3c4ec69d47af38a3e246ce
2026-03-14 09:56:26.614240 | controller |       == e32ab6f2cb3c4ec69d47af38a3e246ce ']'\n    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614245 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:56:26.614251 | controller |       \   logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.614255 | controller |       ']'\n    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | ++ base64 -d\n
2026-03-14 09:56:26.614260 | controller |       \   logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.614264 | controller |       secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n    logger.go:42:
2026-03-14 09:56:26.614268 | controller |       09:52:22 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1\n    logger.go:42:
2026-03-14 09:56:26.614273 | controller |       09:52:22 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:56:26.614277 | controller |       09:52:22 | watcher/1-deploy-with-defaults | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n
2026-03-14 09:56:26.614281 | controller |       \   logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | ++ base64 -d\n    logger.go:42:
2026-03-14 09:56:26.614286 | controller |       09:52:22 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret
2026-03-14 09:56:26.614295 | controller |       watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42:
2026-03-14 09:56:26.614299 | controller |       09:52:22 | watcher/1-deploy-with-defaults | + '[' 2 == 2 ']'\n    logger.go:42:
2026-03-14 09:56:26.614303 | controller |       09:52:22 | watcher/1-deploy-with-defaults | ++ base64 -d\n    logger.go:42: 09:52:22
2026-03-14 09:56:26.614308 | controller |       | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.614312 | controller |       -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614316 | controller |       | ++ grep -c 'region_name = regionOne'\n    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614321 | controller |       | + '[' 9 -ge 1 ']'\n    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614325 | controller |       | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n
2026-03-14 09:56:26.614329 | controller |       \   logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | + '[' '' == '' ']'\n
2026-03-14 09:56:26.614334 | controller |       \   logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | + exit 0\n    logger.go:42:
2026-03-14 09:56:26.614338 | controller |       09:52:23 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:56:26.614342 | controller |       \       oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.614346 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc exec
2026-03-14 09:56:26.614351 | controller |       -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.614355 | controller |       -c Type -c ID | grep watcher| awk '{print $1}')\n        [ $(oc get -n watcher-kuttl-default
2026-03-14 09:56:26.614359 | controller |       keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [
2026-03-14 09:56:26.614363 | controller |       -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\"
2026-03-14 09:56:26.614368 | controller |       ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.614372 | controller |       -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n        [ \"$(oc
2026-03-14 09:56:26.614376 | controller |       get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64
2026-03-14 09:56:26.614381 | controller |       -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2
2026-03-14 09:56:26.614385 | controller |       ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.614389 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\"
2026-03-14 09:56:26.614395 | controller |       -ge 1 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:56:26.614399 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:56:26.614404 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:56:26.614413 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:56:26.614417 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:56:26.614422 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:56:26.614426 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:56:26.614430 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:56:26.614435 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:56:26.614439 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:52:23 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614445 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.614450 | controller |       value -c Name -c Type\n    logger.go:42: 09:52:23 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614454 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:52:25 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614458 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:52:25 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614462 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.614467 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:52:25 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614471 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:52:25 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614475 | controller |       | ++ grep watcher\n    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults |
2026-03-14 09:56:26.614480 | controller |       + SERVICEID=e32ab6f2cb3c4ec69d47af38a3e246ce\n    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614484 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.614490 | controller |       \   logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults | + '[' e32ab6f2cb3c4ec69d47af38a3e246ce
2026-03-14 09:56:26.614494 | controller |       == e32ab6f2cb3c4ec69d47af38a3e246ce ']'\n    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614499 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:56:26.614503 | controller |       \   logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.614507 | controller |       ']'\n    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1\n
2026-03-14 09:56:26.614512 | controller |       \   logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults | ++ base64 -d\n    logger.go:42:
2026-03-14 09:56:26.614516 | controller |       09:52:28 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret
2026-03-14 09:56:26.614520 | controller |       watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n    logger.go:42:
2026-03-14 09:56:26.614524 | controller |       09:52:28 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:56:26.614529 | controller |       09:52:28 | watcher/1-deploy-with-defaults | ++ base64 -d\n    logger.go:42: 09:52:28
2026-03-14 09:56:26.614533 | controller |       | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.614537 | controller |       -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614541 | controller |       | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n    logger.go:42:
2026-03-14 09:56:26.614546 | controller |       09:52:28 | watcher/1-deploy-with-defaults | + '[' 2 == 2 ']'\n    logger.go:42:
2026-03-14 09:56:26.614550 | controller |       09:52:28 | watcher/1-deploy-with-defaults | ++ base64 -d\n    logger.go:42: 09:52:28
2026-03-14 09:56:26.614554 | controller |       | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.614559 | controller |       -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614563 | controller |       | ++ grep -c 'region_name = regionOne'\n    logger.go:42: 09:52:29 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614567 | controller |       | + '[' 9 -ge 1 ']'\n    logger.go:42: 09:52:29 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614571 | controller |       | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n
2026-03-14 09:56:26.614576 | controller |       \   logger.go:42: 09:52:29 | watcher/1-deploy-with-defaults | + '[' '' == '' ']'\n
2026-03-14 09:56:26.614580 | controller |       \   logger.go:42: 09:52:29 | watcher/1-deploy-with-defaults | + exit 0\n    logger.go:42:
2026-03-14 09:56:26.614586 | controller |       09:52:30 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:56:26.614590 | controller |       \       oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.614605 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc exec
2026-03-14 09:56:26.614616 | controller |       -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.614621 | controller |       -c Type -c ID | grep watcher| awk '{print $1}')\n        [ $(oc get -n watcher-kuttl-default
2026-03-14 09:56:26.614625 | controller |       keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [
2026-03-14 09:56:26.614630 | controller |       -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\"
2026-03-14 09:56:26.614634 | controller |       ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.614638 | controller |       -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n        [ \"$(oc
2026-03-14 09:56:26.614642 | controller |       get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64
2026-03-14 09:56:26.614647 | controller |       -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2
2026-03-14 09:56:26.614653 | controller |       ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.614657 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\"
2026-03-14 09:56:26.614662 | controller |       -ge 1 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:56:26.614666 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:56:26.614670 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:56:26.614675 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:56:26.614679 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:56:26.614683 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:56:26.614687 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:56:26.614692 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:56:26.614696 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:56:26.614700 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:52:30 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614705 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.614709 | controller |       value -c Name -c Type\n    logger.go:42: 09:52:30 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614713 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:52:32 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614717 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:52:32 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614722 | controller |       | ++ grep watcher\n    logger.go:42: 09:52:32 | watcher/1-deploy-with-defaults |
2026-03-14 09:56:26.614726 | controller |       ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.614730 | controller |       value -c Name -c Type -c ID\n    logger.go:42: 09:52:32 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614735 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:52:34 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614739 | controller |       | + SERVICEID=e32ab6f2cb3c4ec69d47af38a3e246ce\n    logger.go:42: 09:52:34 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614743 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.614749 | controller |       \   logger.go:42: 09:52:34 | watcher/1-deploy-with-defaults | + '[' e32ab6f2cb3c4ec69d47af38a3e246ce
2026-03-14 09:56:26.614754 | controller |       == e32ab6f2cb3c4ec69d47af38a3e246ce ']'\n    logger.go:42: 09:52:34 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614758 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:56:26.614762 | controller |       \   logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.614767 | controller |       ']'\n    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1\n
2026-03-14 09:56:26.614771 | controller |       \   logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.614775 | controller |       secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n    logger.go:42:
2026-03-14 09:56:26.614779 | controller |       09:52:35 | watcher/1-deploy-with-defaults | ++ base64 -d\n    logger.go:42: 09:52:35
2026-03-14 09:56:26.614784 | controller |       | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n    logger.go:42: 09:52:35
2026-03-14 09:56:26.614788 | controller |       | watcher/1-deploy-with-defaults | ++ base64 -d\n    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614792 | controller |       | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n    logger.go:42:
2026-03-14 09:56:26.614797 | controller |       09:52:35 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret
2026-03-14 09:56:26.614806 | controller |       watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42:
2026-03-14 09:56:26.614811 | controller |       09:52:35 | watcher/1-deploy-with-defaults | + '[' 2 == 2 ']'\n    logger.go:42:
2026-03-14 09:56:26.614815 | controller |       09:52:35 | watcher/1-deploy-with-defaults | ++ base64 -d\n    logger.go:42: 09:52:35
2026-03-14 09:56:26.614820 | controller |       | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.614824 | controller |       -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614828 | controller |       | ++ grep -c 'region_name = regionOne'\n    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614833 | controller |       | + '[' 9 -ge 1 ']'\n    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults
2026-03-14 09:56:26.614837 | controller |       | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n
2026-03-14 09:56:26.614843 | controller |       \   logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | + '[' '' == '' ']'\n
2026-03-14 09:56:26.614849 | controller |       \   logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | + exit 0\n    logger.go:42:
2026-03-14 09:56:26.614855 | controller |       09:52:35 | watcher/1-deploy-with-defaults | test step completed 1-deploy-with-defaults\n
2026-03-14 09:56:26.614861 | controller |       \   logger.go:42: 09:52:35 | watcher/2-cleanup-watcher | starting test step 2-cleanup-watcher\n
2026-03-14 09:56:26.614867 | controller |       \   logger.go:42: 09:52:35 | watcher/2-cleanup-watcher | test step completed 2-cleanup-watcher\n
2026-03-14 09:56:26.614873 | controller |       \   logger.go:42: 09:52:35 | watcher/3-precreate-mariadbaccount | starting test
2026-03-14 09:56:26.614878 | controller |       step 3-precreate-mariadbaccount\n    logger.go:42: 09:52:35 | watcher/3-precreate-mariadbaccount
2026-03-14 09:56:26.614882 | controller |       | MariaDBAccount:watcher-kuttl-default/watcher-precreated created\n    logger.go:42:
2026-03-14 09:56:26.614886 | controller |       09:52:35 | watcher/3-precreate-mariadbaccount | test step completed 3-precreate-mariadbaccount\n
2026-03-14 09:56:26.614890 | controller |       \   logger.go:42: 09:52:35 | watcher/4-deploy-with-precreated-account | starting
2026-03-14 09:56:26.614895 | controller |       test step 4-deploy-with-precreated-account\n    logger.go:42: 09:52:35 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.614903 | controller |       | Secret:watcher-kuttl-default/custom-prometheus-config created\n    logger.go:42:
2026-03-14 09:56:26.614909 | controller |       09:52:35 | watcher/4-deploy-with-precreated-account | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:56:26.614913 | controller |       created\n    logger.go:42: 09:52:35 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.614918 | controller |       running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.614922 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.614926 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.614931 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.614947 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:56:26.614953 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:56:26.614959 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:56:26.614964 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:56:26.614969 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.614974 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:56:26.614979 | controller |       \   logger.go:42: 09:52:35 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.614984 | controller |       watcher-kuttl-default\n    logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.614989 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.614994 | controller |       \   logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.614999 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.615004 | controller |       \   logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.615009 | controller |       -1\n    logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.615014 | controller |       -v '^$'\n    logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.615019 | controller |       + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615033 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615038 | controller |       | ++ grep -c '^# Global config'\n    logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615043 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n
2026-03-14 09:56:26.615048 | controller |       \   logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account | error: Internal
2026-03-14 09:56:26.615053 | controller |       error occurred: error executing command in container: container is not created or
2026-03-14 09:56:26.615058 | controller |       running\n    logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.615063 | controller |       ++ echo\n    logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.615068 | controller |       + '[' 0 == 1 ']'\n    logger.go:42: 09:52:37 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615073 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.615078 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.615085 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.615091 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.615096 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:56:26.615101 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:56:26.615106 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:56:26.615111 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:56:26.615116 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.615121 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:56:26.615126 | controller |       \   logger.go:42: 09:52:37 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.615130 | controller |       watcher-kuttl-default\n    logger.go:42: 09:52:38 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615138 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.615143 | controller |       \   logger.go:42: 09:52:38 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.615148 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.615153 | controller |       \   logger.go:42: 09:52:38 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.615158 | controller |       -1\n    logger.go:42: 09:52:38 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.615163 | controller |       -v '^$'\n    logger.go:42: 09:52:38 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.615169 | controller |       + APIPOD=\n    logger.go:42: 09:52:39 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615174 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.615179 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.615184 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.615189 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.615195 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:56:26.615199 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:56:26.615203 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:56:26.615207 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:56:26.615212 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.615216 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:56:26.615220 | controller |       \   logger.go:42: 09:52:39 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.615224 | controller |       watcher-kuttl-default\n    logger.go:42: 09:52:39 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615235 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.615242 | controller |       \   logger.go:42: 09:52:39 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.615247 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.615251 | controller |       \   logger.go:42: 09:52:39 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.615255 | controller |       -1\n    logger.go:42: 09:52:39 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.615260 | controller |       -v '^$'\n    logger.go:42: 09:52:39 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.615264 | controller |       + APIPOD=\n    logger.go:42: 09:52:40 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615268 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.615273 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.615277 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.615281 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.615286 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:56:26.615290 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:56:26.615294 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:56:26.615301 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:56:26.615306 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.615310 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:56:26.615314 | controller |       \   logger.go:42: 09:52:40 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.615319 | controller |       watcher-kuttl-default\n    logger.go:42: 09:52:40 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615323 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.615327 | controller |       \   logger.go:42: 09:52:40 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.615331 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.615336 | controller |       \   logger.go:42: 09:52:40 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.615340 | controller |       -1\n    logger.go:42: 09:52:40 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.615344 | controller |       -v '^$'\n    logger.go:42: 09:52:41 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.615349 | controller |       + APIPOD=\n    logger.go:42: 09:52:42 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615353 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.615357 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.615361 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.615366 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.615370 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:56:26.615376 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:56:26.615380 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:56:26.615385 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:56:26.615391 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.615395 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:56:26.615400 | controller |       \   logger.go:42: 09:52:42 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.615404 | controller |       watcher-kuttl-default\n    logger.go:42: 09:52:42 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615408 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.615412 | controller |       \   logger.go:42: 09:52:42 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.615423 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.615427 | controller |       \   logger.go:42: 09:52:42 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.615432 | controller |       -1\n    logger.go:42: 09:52:42 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.615436 | controller |       -v '^$'\n    logger.go:42: 09:52:42 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.615440 | controller |       + APIPOD=\n    logger.go:42: 09:52:43 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615445 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.615449 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.615453 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.615458 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.615462 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:56:26.615466 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:56:26.615470 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:56:26.615475 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:56:26.615479 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.615483 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:56:26.615488 | controller |       \   logger.go:42: 09:52:43 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.615492 | controller |       watcher-kuttl-default\n    logger.go:42: 09:52:43 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615496 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.615501 | controller |       \   logger.go:42: 09:52:43 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.615505 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.615509 | controller |       \   logger.go:42: 09:52:43 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.615514 | controller |       -1\n    logger.go:42: 09:52:43 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.615518 | controller |       -v '^$'\n    logger.go:42: 09:52:43 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.615522 | controller |       + APIPOD=\n    logger.go:42: 09:52:45 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615528 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.615532 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.615537 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.615541 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.615545 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:56:26.615550 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:56:26.615554 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:56:26.615558 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:56:26.615563 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.615567 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:56:26.615573 | controller |       \   logger.go:42: 09:52:45 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.615577 | controller |       watcher-kuttl-default\n    logger.go:42: 09:52:45 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615582 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.615586 | controller |       \   logger.go:42: 09:52:45 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.615590 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.615607 | controller |       \   logger.go:42: 09:52:45 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.615622 | controller |       -1\n    logger.go:42: 09:52:45 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.615627 | controller |       -v '^$'\n    logger.go:42: 09:52:45 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.615631 | controller |       + APIPOD=\n    logger.go:42: 09:52:46 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615636 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.615640 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.615644 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.615649 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.615653 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:56:26.615657 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:56:26.615662 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:56:26.615666 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:56:26.615670 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.615675 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:56:26.615679 | controller |       \   logger.go:42: 09:52:46 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.615683 | controller |       watcher-kuttl-default\n    logger.go:42: 09:52:46 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615689 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.615694 | controller |       \   logger.go:42: 09:52:46 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.615698 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.615702 | controller |       \   logger.go:42: 09:52:46 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.615707 | controller |       -1\n    logger.go:42: 09:52:46 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.615711 | controller |       -v '^$'\n    logger.go:42: 09:52:46 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.615715 | controller |       + APIPOD=\n    logger.go:42: 09:52:47 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615719 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.615724 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.615728 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.615732 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.615737 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:56:26.615741 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:56:26.615745 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:56:26.615750 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:56:26.615754 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.615760 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:56:26.615764 | controller |       \   logger.go:42: 09:52:47 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.615769 | controller |       watcher-kuttl-default\n    logger.go:42: 09:52:47 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615773 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.615777 | controller |       \   logger.go:42: 09:52:47 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.615781 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.615786 | controller |       \   logger.go:42: 09:52:47 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.615790 | controller |       -1\n    logger.go:42: 09:52:47 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.615794 | controller |       -v '^$'\n    logger.go:42: 09:52:48 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.615798 | controller |       + APIPOD=\n    logger.go:42: 09:52:49 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615811 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.615816 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.615820 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.615824 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.615829 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:56:26.615833 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:56:26.615839 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:56:26.615843 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:56:26.615848 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.615852 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:56:26.615856 | controller |       \   logger.go:42: 09:52:49 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.615861 | controller |       watcher-kuttl-default\n    logger.go:42: 09:52:49 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615865 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.615869 | controller |       \   logger.go:42: 09:52:49 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.615873 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.615878 | controller |       \   logger.go:42: 09:52:49 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.615882 | controller |       -1\n    logger.go:42: 09:52:49 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.615886 | controller |       -v '^$'\n    logger.go:42: 09:52:49 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.615891 | controller |       + APIPOD=\n    logger.go:42: 09:52:50 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615895 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.615899 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.615903 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.615908 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.615912 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:56:26.615916 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:56:26.615921 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:56:26.615925 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:56:26.615929 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.615934 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:56:26.615939 | controller |       \   logger.go:42: 09:52:50 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.615945 | controller |       watcher-kuttl-default\n    logger.go:42: 09:52:50 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615951 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.615957 | controller |       \   logger.go:42: 09:52:50 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.615962 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.615966 | controller |       \   logger.go:42: 09:52:50 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.615972 | controller |       -1\n    logger.go:42: 09:52:50 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.615981 | controller |       -v '^$'\n    logger.go:42: 09:52:50 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.615987 | controller |       + APIPOD=\n    logger.go:42: 09:52:51 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.615991 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:56:26.615995 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:56:26.616006 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:56:26.616011 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.616015 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:56:26.616020 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:56:26.616024 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:56:26.616028 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:56:26.616033 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.616037 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:56:26.616044 | controller |       \   logger.go:42: 09:52:51 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.616048 | controller |       watcher-kuttl-default\n    logger.go:42: 09:52:52 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616052 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.616057 | controller |       \   logger.go:42: 09:52:52 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.616061 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.616065 | controller |       \   logger.go:42: 09:52:52 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.616070 | controller |       -1\n    logger.go:42: 09:52:52 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.616074 | controller |       -v '^$'\n    logger.go:42: 09:52:52 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.616078 | controller |       + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:52:52 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616082 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:52:52 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616087 | controller |       | ++ grep -c '^# Global config'\n    logger.go:42: 09:52:52 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616091 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n
2026-03-14 09:56:26.616095 | controller |       \   logger.go:42: 09:52:52 | watcher/4-deploy-with-precreated-account | error: unable
2026-03-14 09:56:26.616099 | controller |       to upgrade connection: container not found (\"watcher-api\")\n    logger.go:42:
2026-03-14 09:56:26.616104 | controller |       09:52:52 | watcher/4-deploy-with-precreated-account | ++ echo\n    logger.go:42:
2026-03-14 09:56:26.616108 | controller |       09:52:52 | watcher/4-deploy-with-precreated-account | + '[' 0 == 1 ']'\n    logger.go:42:
2026-03-14 09:56:26.616112 | controller |       09:52:53 | watcher/4-deploy-with-precreated-account | running command: [sh -c set
2026-03-14 09:56:26.616117 | controller |       -euxo pipefail\n        oc project watcher-kuttl-default\n        APIPOD=$(oc get
2026-03-14 09:56:26.616121 | controller |       pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep
2026-03-14 09:56:26.616125 | controller |       -v ^$|head -1)\n        if [ -n \"${APIPOD}\" ]; then\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.616131 | controller |       -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.616135 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:56:26.616140 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:56:26.616144 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:56:26.616148 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:56:26.616153 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.616157 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:56:26.616161 | controller |       \   logger.go:42: 09:52:53 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.616165 | controller |       watcher-kuttl-default\n    logger.go:42: 09:52:53 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616170 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.616174 | controller |       \   logger.go:42: 09:52:53 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.616184 | controller |       -1\n    logger.go:42: 09:52:53 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.616188 | controller |       -v '^$'\n    logger.go:42: 09:52:53 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.616193 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.616197 | controller |       \   logger.go:42: 09:52:53 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n
2026-03-14 09:56:26.616201 | controller |       \   logger.go:42: 09:52:53 | watcher/4-deploy-with-precreated-account | + '[' -n
2026-03-14 09:56:26.616205 | controller |       watcher-kuttl-api-0 ']'\n    logger.go:42: 09:52:53 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616210 | controller |       | ++ grep -c '^# Global config'\n    logger.go:42: 09:52:53 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616214 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n
2026-03-14 09:56:26.616218 | controller |       \   logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.616223 | controller |       '#' Global config\n    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616227 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616231 | controller |       | ++ grep -c '^# Service config'\n    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616235 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n
2026-03-14 09:56:26.616240 | controller |       \   logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.616246 | controller |       '#' Service config\n    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616250 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616254 | controller |       | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n
2026-03-14 09:56:26.616259 | controller |       \   logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account | +++ oc rsh
2026-03-14 09:56:26.616263 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.616269 | controller |       \   logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.616274 | controller |       '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.616278 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.616282 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.616287 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_test:bcdc991c7f059e29ee45381194286841@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.616291 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.616295 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.616299 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.616304 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.616308 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.616312 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.616316 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.616325 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.616329 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.616333 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.616338 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.616342 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.616346 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.616350 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.616355 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.616359 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.616369 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.616374 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.616378 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.616382 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]'
2026-03-14 09:56:26.616387 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.616391 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.616395 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.616400 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.616404 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.616408 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.616412 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.616419 | controller |       period = 900\n    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616423 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616428 | controller |       | ++ grep -czPo 'TimeOut 80'\n    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616434 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n
2026-03-14 09:56:26.616438 | controller |       \   logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.616443 | controller |       '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '<VirtualHost'
2026-03-14 09:56:26.616447 | controller |       '*:9322>' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot
2026-03-14 09:56:26.616451 | controller |       DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.616455 | controller |       80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.616460 | controller |       '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.616464 | controller |       AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog /dev/stdout
2026-03-14 09:56:26.616468 | controller |       ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout
2026-03-14 09:56:26.616472 | controller |       proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI
2026-03-14 09:56:26.616477 | controller |       configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal
2026-03-14 09:56:26.616481 | controller |       group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias
2026-03-14 09:56:26.616485 | controller |       / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>' '#' public vhost watcher-public.watcher-kuttl-default.svc
2026-03-14 09:56:26.616490 | controller |       configuration '<VirtualHost' '*:9322>' ServerName watcher-public.watcher-kuttl-default.svc
2026-03-14 09:56:26.616494 | controller |       '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the
2026-03-14 09:56:26.616498 | controller |       watcher-api TimeOut 80 '##' Directories, there should at least be a declaration
2026-03-14 09:56:26.616502 | controller |       for /var/www/cgi-bin '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks
2026-03-14 09:56:26.616507 | controller |       +MultiViews AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog
2026-03-14 09:56:26.616511 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded'
2026-03-14 09:56:26.616515 | controller |       CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel
2026-03-14 09:56:26.616519 | controller |       debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess
2026-03-14 09:56:26.616524 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:56:26.616528 | controller |       public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>'\n    logger.go:42:
2026-03-14 09:56:26.616532 | controller |       09:52:54 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:56:26.616536 | controller |       09:52:55 | watcher/4-deploy-with-precreated-account | running command: [sh -c set
2026-03-14 09:56:26.616541 | controller |       -euxo pipefail\n        oc project watcher-kuttl-default\n        APIPOD=$(oc get
2026-03-14 09:56:26.616545 | controller |       pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep
2026-03-14 09:56:26.616549 | controller |       -v ^$|head -1)\n        if [ -n \"${APIPOD}\" ]; then\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.616559 | controller |       -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.616563 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:56:26.616570 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:56:26.616574 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:56:26.616578 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:56:26.616583 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.616587 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:56:26.616591 | controller |       \   logger.go:42: 09:52:55 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.616609 | controller |       watcher-kuttl-default\n    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616615 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.616620 | controller |       \   logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.616624 | controller |       -1\n    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | ++ oc
2026-03-14 09:56:26.616628 | controller |       get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.616633 | controller |       \   logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.616637 | controller |       -v '^$'\n    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.616641 | controller |       + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616647 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616651 | controller |       | ++ grep -c '^# Global config'\n    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616656 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n
2026-03-14 09:56:26.616660 | controller |       \   logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.616664 | controller |       '#' Global config\n    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616669 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616673 | controller |       | ++ grep -c '^# Service config'\n    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616677 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n
2026-03-14 09:56:26.616682 | controller |       \   logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.616686 | controller |       '#' Service config\n    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616690 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616695 | controller |       | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n
2026-03-14 09:56:26.616699 | controller |       \   logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | +++ oc rsh
2026-03-14 09:56:26.616703 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.616707 | controller |       \   logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.616712 | controller |       '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.616718 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.616722 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.616727 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_test:bcdc991c7f059e29ee45381194286841@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.616731 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.616735 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.616740 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.616750 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.616755 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.616759 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.616763 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.616768 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.616772 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.616776 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.616780 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.616785 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.616789 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.616793 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.616798 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.616802 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.616806 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.616811 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.616815 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.616819 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]'
2026-03-14 09:56:26.616823 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.616828 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.616832 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.616836 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.616841 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.616845 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.616849 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.616854 | controller |       period = 900\n    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616860 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616864 | controller |       | ++ grep -czPo 'TimeOut 80'\n    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.616869 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n
2026-03-14 09:56:26.616873 | controller |       \   logger.go:42: 09:52:57 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.616877 | controller |       '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '<VirtualHost'
2026-03-14 09:56:26.616883 | controller |       '*:9322>' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot
2026-03-14 09:56:26.616887 | controller |       DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.616892 | controller |       80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.616896 | controller |       '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.616900 | controller |       AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog /dev/stdout
2026-03-14 09:56:26.616904 | controller |       ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout
2026-03-14 09:56:26.616909 | controller |       proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI
2026-03-14 09:56:26.616913 | controller |       configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal
2026-03-14 09:56:26.616917 | controller |       group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias
2026-03-14 09:56:26.616922 | controller |       / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>' '#' public vhost watcher-public.watcher-kuttl-default.svc
2026-03-14 09:56:26.616926 | controller |       configuration '<VirtualHost' '*:9322>' ServerName watcher-public.watcher-kuttl-default.svc
2026-03-14 09:56:26.616930 | controller |       '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the
2026-03-14 09:56:26.616934 | controller |       watcher-api TimeOut 80 '##' Directories, there should at least be a declaration
2026-03-14 09:56:26.616944 | controller |       for /var/www/cgi-bin '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks
2026-03-14 09:56:26.616948 | controller |       +MultiViews AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog
2026-03-14 09:56:26.616953 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded'
2026-03-14 09:56:26.616957 | controller |       CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel
2026-03-14 09:56:26.616961 | controller |       debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess
2026-03-14 09:56:26.616966 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:56:26.616970 | controller |       public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>'\n    logger.go:42:
2026-03-14 09:56:26.616976 | controller |       09:52:57 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:56:26.616980 | controller |       09:52:58 | watcher/4-deploy-with-precreated-account | running command: [sh -c set
2026-03-14 09:56:26.616984 | controller |       -euxo pipefail\n        oc project watcher-kuttl-default\n        APIPOD=$(oc get
2026-03-14 09:56:26.616989 | controller |       pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep
2026-03-14 09:56:26.616993 | controller |       -v ^$|head -1)\n        if [ -n \"${APIPOD}\" ]; then\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.616997 | controller |       -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.617002 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:56:26.617006 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:56:26.617010 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:56:26.617016 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:56:26.617021 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.617025 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:56:26.617029 | controller |       \   logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.617033 | controller |       watcher-kuttl-default\n    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617038 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.617042 | controller |       \   logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.617046 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.617051 | controller |       \   logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.617055 | controller |       -v '^$'\n    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.617059 | controller |       ++ head -1\n    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617063 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617068 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617072 | controller |       | ++ grep -c '^# Global config'\n    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617076 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n
2026-03-14 09:56:26.617081 | controller |       \   logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.617085 | controller |       '#' Global config\n    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617089 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617093 | controller |       | ++ grep -c '^# Service config'\n    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617098 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n
2026-03-14 09:56:26.617102 | controller |       \   logger.go:42: 09:52:59 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.617106 | controller |       '#' Service config\n    logger.go:42: 09:52:59 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617111 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:52:59 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617115 | controller |       | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n
2026-03-14 09:56:26.617124 | controller |       \   logger.go:42: 09:52:59 | watcher/4-deploy-with-precreated-account | +++ oc rsh
2026-03-14 09:56:26.617129 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.617133 | controller |       \   logger.go:42: 09:52:59 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.617137 | controller |       '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.617142 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.617149 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.617154 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_test:bcdc991c7f059e29ee45381194286841@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.617158 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.617162 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.617167 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.617171 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.617175 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.617180 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.617184 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.617188 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.617193 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.617197 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.617201 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.617208 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.617214 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.617220 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.617226 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.617232 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.617238 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.617242 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.617246 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.617251 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]'
2026-03-14 09:56:26.617255 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.617259 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.617264 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.617268 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.617272 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.617276 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.617281 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.617285 | controller |       period = 900\n    logger.go:42: 09:52:59 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617289 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:52:59 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617294 | controller |       | ++ grep -czPo 'TimeOut 80'\n    logger.go:42: 09:52:59 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617300 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n
2026-03-14 09:56:26.617305 | controller |       \   logger.go:42: 09:52:59 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.617309 | controller |       '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '<VirtualHost'
2026-03-14 09:56:26.617320 | controller |       '*:9322>' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot
2026-03-14 09:56:26.617324 | controller |       DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.617328 | controller |       80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.617333 | controller |       '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.617337 | controller |       AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog /dev/stdout
2026-03-14 09:56:26.617341 | controller |       ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout
2026-03-14 09:56:26.617346 | controller |       proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI
2026-03-14 09:56:26.617350 | controller |       configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal
2026-03-14 09:56:26.617357 | controller |       group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias
2026-03-14 09:56:26.617361 | controller |       / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>' '#' public vhost watcher-public.watcher-kuttl-default.svc
2026-03-14 09:56:26.617365 | controller |       configuration '<VirtualHost' '*:9322>' ServerName watcher-public.watcher-kuttl-default.svc
2026-03-14 09:56:26.617373 | controller |       '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the
2026-03-14 09:56:26.617377 | controller |       watcher-api TimeOut 80 '##' Directories, there should at least be a declaration
2026-03-14 09:56:26.617382 | controller |       for /var/www/cgi-bin '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks
2026-03-14 09:56:26.617386 | controller |       +MultiViews AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog
2026-03-14 09:56:26.617390 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded'
2026-03-14 09:56:26.617394 | controller |       CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel
2026-03-14 09:56:26.617399 | controller |       debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess
2026-03-14 09:56:26.617403 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:56:26.617407 | controller |       public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>'\n    logger.go:42:
2026-03-14 09:56:26.617412 | controller |       09:52:59 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:56:26.617416 | controller |       09:53:00 | watcher/4-deploy-with-precreated-account | running command: [sh -c set
2026-03-14 09:56:26.617420 | controller |       -euxo pipefail\n        oc project watcher-kuttl-default\n        APIPOD=$(oc get
2026-03-14 09:56:26.617426 | controller |       pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep
2026-03-14 09:56:26.617430 | controller |       -v ^$|head -1)\n        if [ -n \"${APIPOD}\" ]; then\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.617435 | controller |       -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.617439 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:56:26.617443 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:56:26.617447 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:56:26.617452 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:56:26.617458 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.617462 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:56:26.617467 | controller |       \   logger.go:42: 09:53:00 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.617471 | controller |       watcher-kuttl-default\n    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617475 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.617480 | controller |       \   logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.617484 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.617488 | controller |       \   logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.617492 | controller |       -1\n    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.617497 | controller |       -v '^$'\n    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.617501 | controller |       + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617505 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617515 | controller |       | ++ grep -c '^# Global config'\n    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617519 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n
2026-03-14 09:56:26.617523 | controller |       \   logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.617528 | controller |       '#' Global config\n    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617532 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617536 | controller |       | ++ grep -c '^# Service config'\n    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617541 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n
2026-03-14 09:56:26.617545 | controller |       \   logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.617551 | controller |       '#' Service config\n    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617556 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617560 | controller |       | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n
2026-03-14 09:56:26.617564 | controller |       \   logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | +++ oc rsh
2026-03-14 09:56:26.617569 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.617573 | controller |       \   logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.617577 | controller |       '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.617581 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.617586 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.617590 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_test:bcdc991c7f059e29ee45381194286841@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.617608 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.617614 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.617619 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.617623 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.617627 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.617632 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.617636 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.617640 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.617645 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.617649 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.617653 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.617657 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.617662 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.617666 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.617670 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.617675 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.617679 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.617683 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.617687 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.617692 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]'
2026-03-14 09:56:26.617698 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.617703 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.617713 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.617718 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.617723 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.617727 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.617731 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.617736 | controller |       period = 900\n    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617740 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617746 | controller |       | ++ grep -czPo 'TimeOut 80'\n    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617750 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n
2026-03-14 09:56:26.617755 | controller |       \   logger.go:42: 09:53:02 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.617761 | controller |       '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '<VirtualHost'
2026-03-14 09:56:26.617765 | controller |       '*:9322>' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot
2026-03-14 09:56:26.617769 | controller |       DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.617774 | controller |       80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.617778 | controller |       '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.617782 | controller |       AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog /dev/stdout
2026-03-14 09:56:26.617786 | controller |       ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout
2026-03-14 09:56:26.617791 | controller |       proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI
2026-03-14 09:56:26.617795 | controller |       configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal
2026-03-14 09:56:26.617799 | controller |       group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias
2026-03-14 09:56:26.617804 | controller |       / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>' '#' public vhost watcher-public.watcher-kuttl-default.svc
2026-03-14 09:56:26.617808 | controller |       configuration '<VirtualHost' '*:9322>' ServerName watcher-public.watcher-kuttl-default.svc
2026-03-14 09:56:26.617812 | controller |       '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the
2026-03-14 09:56:26.617816 | controller |       watcher-api TimeOut 80 '##' Directories, there should at least be a declaration
2026-03-14 09:56:26.617821 | controller |       for /var/www/cgi-bin '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks
2026-03-14 09:56:26.617825 | controller |       +MultiViews AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog
2026-03-14 09:56:26.617829 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded'
2026-03-14 09:56:26.617833 | controller |       CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel
2026-03-14 09:56:26.617838 | controller |       debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess
2026-03-14 09:56:26.617842 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:56:26.617846 | controller |       public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>'\n    logger.go:42:
2026-03-14 09:56:26.617851 | controller |       09:53:02 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:56:26.617855 | controller |       09:53:03 | watcher/4-deploy-with-precreated-account | running command: [sh -c set
2026-03-14 09:56:26.617859 | controller |       -euxo pipefail\n        oc project watcher-kuttl-default\n        APIPOD=$(oc get
2026-03-14 09:56:26.617863 | controller |       pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep
2026-03-14 09:56:26.617868 | controller |       -v ^$|head -1)\n        if [ -n \"${APIPOD}\" ]; then\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.617872 | controller |       -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.617876 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:56:26.617880 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:56:26.617885 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:56:26.617891 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:56:26.617900 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.617906 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:56:26.617911 | controller |       \   logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.617915 | controller |       watcher-kuttl-default\n    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617919 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.617923 | controller |       \   logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.617928 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.617932 | controller |       \   logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.617936 | controller |       -1\n    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.617940 | controller |       -v '^$'\n    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.617945 | controller |       + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617949 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617953 | controller |       | ++ grep -c '^# Global config'\n    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617958 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n
2026-03-14 09:56:26.617962 | controller |       \   logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.617968 | controller |       '#' Global config\n    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617972 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617977 | controller |       | ++ grep -c '^# Service config'\n    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617981 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n
2026-03-14 09:56:26.617985 | controller |       \   logger.go:42: 09:53:04 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.617989 | controller |       '#' Service config\n    logger.go:42: 09:53:04 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617994 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:53:04 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.617998 | controller |       | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n
2026-03-14 09:56:26.618002 | controller |       \   logger.go:42: 09:53:04 | watcher/4-deploy-with-precreated-account | +++ oc rsh
2026-03-14 09:56:26.618007 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.618011 | controller |       \   logger.go:42: 09:53:04 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.618015 | controller |       '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.618020 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.618024 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.618028 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_test:bcdc991c7f059e29ee45381194286841@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.618033 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.618038 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.618043 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.618047 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.618051 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.618056 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.618060 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.618064 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.618068 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.618079 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.618084 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.618088 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.618093 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.618097 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.618101 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.618106 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.618110 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.618116 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.618122 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.618128 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]'
2026-03-14 09:56:26.618134 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.618140 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.618146 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.618150 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.618155 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.618159 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.618163 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.618168 | controller |       period = 900\n    logger.go:42: 09:53:04 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618172 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:53:04 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618176 | controller |       | ++ grep -czPo 'TimeOut 80'\n    logger.go:42: 09:53:04 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618181 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n
2026-03-14 09:56:26.618185 | controller |       \   logger.go:42: 09:53:04 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.618189 | controller |       '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '<VirtualHost'
2026-03-14 09:56:26.618196 | controller |       '*:9322>' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot
2026-03-14 09:56:26.618200 | controller |       DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.618204 | controller |       80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.618209 | controller |       '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.618213 | controller |       AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog /dev/stdout
2026-03-14 09:56:26.618217 | controller |       ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout
2026-03-14 09:56:26.618222 | controller |       proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI
2026-03-14 09:56:26.618226 | controller |       configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal
2026-03-14 09:56:26.618230 | controller |       group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias
2026-03-14 09:56:26.618234 | controller |       / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>' '#' public vhost watcher-public.watcher-kuttl-default.svc
2026-03-14 09:56:26.618239 | controller |       configuration '<VirtualHost' '*:9322>' ServerName watcher-public.watcher-kuttl-default.svc
2026-03-14 09:56:26.618243 | controller |       '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the
2026-03-14 09:56:26.618247 | controller |       watcher-api TimeOut 80 '##' Directories, there should at least be a declaration
2026-03-14 09:56:26.618251 | controller |       for /var/www/cgi-bin '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks
2026-03-14 09:56:26.618258 | controller |       +MultiViews AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog
2026-03-14 09:56:26.618262 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded'
2026-03-14 09:56:26.618266 | controller |       CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel
2026-03-14 09:56:26.618270 | controller |       debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess
2026-03-14 09:56:26.618275 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:56:26.618285 | controller |       public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>'\n    logger.go:42:
2026-03-14 09:56:26.618290 | controller |       09:53:04 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:56:26.618294 | controller |       09:53:05 | watcher/4-deploy-with-precreated-account | running command: [sh -c set
2026-03-14 09:56:26.618299 | controller |       -euxo pipefail\n        oc project watcher-kuttl-default\n        APIPOD=$(oc get
2026-03-14 09:56:26.618303 | controller |       pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep
2026-03-14 09:56:26.618307 | controller |       -v ^$|head -1)\n        if [ -n \"${APIPOD}\" ]; then\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.618312 | controller |       -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.618316 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:56:26.618320 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:56:26.618324 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:56:26.618329 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:56:26.618334 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.618339 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:56:26.618344 | controller |       \   logger.go:42: 09:53:05 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.618352 | controller |       watcher-kuttl-default\n    logger.go:42: 09:53:05 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618357 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.618362 | controller |       \   logger.go:42: 09:53:05 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.618367 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.618371 | controller |       \   logger.go:42: 09:53:05 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.618375 | controller |       -1\n    logger.go:42: 09:53:05 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.618379 | controller |       -v '^$'\n    logger.go:42: 09:53:05 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.618384 | controller |       + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:53:05 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618388 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:53:05 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618392 | controller |       | ++ grep -c '^# Global config'\n    logger.go:42: 09:53:05 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618397 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n
2026-03-14 09:56:26.618401 | controller |       \   logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.618405 | controller |       '#' Global config\n    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618409 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618414 | controller |       | ++ grep -c '^# Service config'\n    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618418 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n
2026-03-14 09:56:26.618422 | controller |       \   logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.618427 | controller |       '#' Service config\n    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618431 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618435 | controller |       | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n
2026-03-14 09:56:26.618439 | controller |       \   logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account | +++ oc rsh
2026-03-14 09:56:26.618444 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.618448 | controller |       \   logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.618452 | controller |       '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.618458 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.618468 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.618472 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_test:bcdc991c7f059e29ee45381194286841@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.618477 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.618481 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.618485 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.618492 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.618496 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.618500 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.618504 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.618509 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.618513 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.618517 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.618521 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.618526 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.618530 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.618536 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.618540 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.618545 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.618549 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.618553 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.618557 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.618562 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]'
2026-03-14 09:56:26.618566 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.618570 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.618574 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.618579 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.618583 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.618587 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.618592 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.618609 | controller |       period = 900\n    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618614 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618618 | controller |       | ++ grep -czPo 'TimeOut 80'\n    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618622 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n
2026-03-14 09:56:26.618626 | controller |       \   logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.618631 | controller |       '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '<VirtualHost'
2026-03-14 09:56:26.618635 | controller |       '*:9322>' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot
2026-03-14 09:56:26.618639 | controller |       DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.618645 | controller |       80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.618650 | controller |       '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.618654 | controller |       AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog /dev/stdout
2026-03-14 09:56:26.618658 | controller |       ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout
2026-03-14 09:56:26.618668 | controller |       proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI
2026-03-14 09:56:26.618673 | controller |       configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal
2026-03-14 09:56:26.618677 | controller |       group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias
2026-03-14 09:56:26.618681 | controller |       / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>' '#' public vhost watcher-public.watcher-kuttl-default.svc
2026-03-14 09:56:26.618686 | controller |       configuration '<VirtualHost' '*:9322>' ServerName watcher-public.watcher-kuttl-default.svc
2026-03-14 09:56:26.618690 | controller |       '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the
2026-03-14 09:56:26.618694 | controller |       watcher-api TimeOut 80 '##' Directories, there should at least be a declaration
2026-03-14 09:56:26.618698 | controller |       for /var/www/cgi-bin '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks
2026-03-14 09:56:26.618703 | controller |       +MultiViews AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog
2026-03-14 09:56:26.618707 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded'
2026-03-14 09:56:26.618711 | controller |       CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel
2026-03-14 09:56:26.618715 | controller |       debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess
2026-03-14 09:56:26.618720 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:56:26.618724 | controller |       public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>'\n    logger.go:42:
2026-03-14 09:56:26.618728 | controller |       09:53:06 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:56:26.618733 | controller |       09:53:07 | watcher/4-deploy-with-precreated-account | running command: [sh -c set
2026-03-14 09:56:26.618737 | controller |       -euxo pipefail\n        oc project watcher-kuttl-default\n        APIPOD=$(oc get
2026-03-14 09:56:26.618741 | controller |       pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep
2026-03-14 09:56:26.618745 | controller |       -v ^$|head -1)\n        if [ -n \"${APIPOD}\" ]; then\n          [ $(echo $(oc rsh
2026-03-14 09:56:26.618750 | controller |       -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.618754 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:56:26.618758 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:56:26.618762 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:56:26.618767 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:56:26.618771 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.618775 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:56:26.618780 | controller |       \   logger.go:42: 09:53:07 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.618784 | controller |       watcher-kuttl-default\n    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618790 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:56:26.618794 | controller |       \   logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.618800 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:56:26.618804 | controller |       \   logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.618809 | controller |       -1\n    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.618813 | controller |       -v '^$'\n    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:56:26.618817 | controller |       + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618821 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618826 | controller |       | ++ grep -c '^# Global config'\n    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618830 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n
2026-03-14 09:56:26.618834 | controller |       \   logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.618839 | controller |       '#' Global config\n    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618844 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618854 | controller |       | ++ grep -c '^# Service config'\n    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618858 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n
2026-03-14 09:56:26.618863 | controller |       \   logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.618867 | controller |       '#' Service config\n    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618871 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.618876 | controller |       | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n
2026-03-14 09:56:26.618880 | controller |       \   logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | +++ oc rsh
2026-03-14 09:56:26.618884 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:56:26.618889 | controller |       \   logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.618893 | controller |       '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:56:26.618900 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.618904 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.618908 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_test:bcdc991c7f059e29ee45381194286841@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:56:26.618913 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:56:26.618917 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.618921 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.618926 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.618932 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.618936 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.618955 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.618961 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.618965 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.618969 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:56:26.618974 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:56:26.618978 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:56:26.618982 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:56:26.618986 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:56:26.618991 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.618995 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.618999 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.619004 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:56:26.619008 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.619012 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]'
2026-03-14 09:56:26.619016 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:56:26.619021 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.619025 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:56:26.619029 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:56:26.619033 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:56:26.619038 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:56:26.619042 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:56:26.619052 | controller |       period = 900\n    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.619056 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.619061 | controller |       | ++ grep -czPo 'TimeOut 80'\n    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account
2026-03-14 09:56:26.619065 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n
2026-03-14 09:56:26.619069 | controller |       \   logger.go:42: 09:53:09 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.619073 | controller |       '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '<VirtualHost'
2026-03-14 09:56:26.619078 | controller |       '*:9322>' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot
2026-03-14 09:56:26.619082 | controller |       DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.619086 | controller |       80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.619090 | controller |       '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.619097 | controller |       AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog /dev/stdout
2026-03-14 09:56:26.619101 | controller |       ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout
2026-03-14 09:56:26.619105 | controller |       proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI
2026-03-14 09:56:26.619110 | controller |       configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal
2026-03-14 09:56:26.619114 | controller |       group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias
2026-03-14 09:56:26.619118 | controller |       / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>' '#' public vhost watcher-public.watcher-kuttl-default.svc
2026-03-14 09:56:26.619123 | controller |       configuration '<VirtualHost' '*:9322>' ServerName watcher-public.watcher-kuttl-default.svc
2026-03-14 09:56:26.619127 | controller |       '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the
2026-03-14 09:56:26.619131 | controller |       watcher-api TimeOut 80 '##' Directories, there should at least be a declaration
2026-03-14 09:56:26.619135 | controller |       for /var/www/cgi-bin '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks
2026-03-14 09:56:26.619140 | controller |       +MultiViews AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog
2026-03-14 09:56:26.619144 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded'
2026-03-14 09:56:26.619148 | controller |       CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel
2026-03-14 09:56:26.619152 | controller |       debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess
2026-03-14 09:56:26.619157 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:56:26.619161 | controller |       public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>'\n    logger.go:42:
2026-03-14 09:56:26.619165 | controller |       09:53:09 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:56:26.619169 | controller |       09:53:09 | watcher/4-deploy-with-precreated-account | test step completed 4-deploy-with-precreated-account\n
2026-03-14 09:56:26.619174 | controller |       \   logger.go:42: 09:53:09 | watcher/5-cleanup-watcher | starting test step 5-cleanup-watcher\n
2026-03-14 09:56:26.619178 | controller |       \   logger.go:42: 09:53:09 | watcher/5-cleanup-watcher | running command: [sh -c
2026-03-14 09:56:26.619182 | controller |       set -ex\n        oc exec -n watcher-kuttl-default openstackclient -- openstack service
2026-03-14 09:56:26.619187 | controller |       list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.619193 | controller |       09:53:09 | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.619199 | controller |       -- openstack service list -f value -c Name -c Type\n    logger.go:42: 09:53:09 |
2026-03-14 09:56:26.619206 | controller |       watcher/5-cleanup-watcher | ++ grep -c '^watcher'\n    logger.go:42: 09:53:12 |
2026-03-14 09:56:26.619211 | controller |       watcher/5-cleanup-watcher | + '[' 1 == 0 ']'\n    logger.go:42: 09:53:13 | watcher/5-cleanup-watcher
2026-03-14 09:56:26.619217 | controller |       | running command: [sh -c set -ex\n        oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.619221 | controller |       -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0
2026-03-14 09:56:26.619225 | controller |       ]\n        ]\n    logger.go:42: 09:53:13 | watcher/5-cleanup-watcher | + oc exec
2026-03-14 09:56:26.619230 | controller |       -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.619234 | controller |       -c Type\n    logger.go:42: 09:53:13 | watcher/5-cleanup-watcher | ++ grep -c '^watcher'\n
2026-03-14 09:56:26.619238 | controller |       \   logger.go:42: 09:53:15 | watcher/5-cleanup-watcher | + '[' 0 == 0 ']'\n    logger.go:42:
2026-03-14 09:56:26.619243 | controller |       09:53:16 | watcher/5-cleanup-watcher | running command: [sh -c set -ex\n        oc
2026-03-14 09:56:26.619255 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.619260 | controller |       -c Name -c Type | [ $(grep -c ^watcher) == 0 ]\n        ]\n    logger.go:42: 09:53:16
2026-03-14 09:56:26.619264 | controller |       | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.619269 | controller |       -- openstack service list -f value -c Name -c Type\n    logger.go:42: 09:53:16 |
2026-03-14 09:56:26.619273 | controller |       watcher/5-cleanup-watcher | ++ grep -c '^watcher'\n    logger.go:42: 09:53:18 |
2026-03-14 09:56:26.619277 | controller |       watcher/5-cleanup-watcher | + '[' 0 == 0 ']'\n    logger.go:42: 09:53:18 | watcher/5-cleanup-watcher
2026-03-14 09:56:26.619282 | controller |       | test step completed 5-cleanup-watcher\n    logger.go:42: 09:53:18 | watcher |
2026-03-14 09:56:26.619286 | controller |       skipping kubernetes event logging\n=== CONT  kuttl/harness/watcher-cinder\n    logger.go:42:
2026-03-14 09:56:26.619290 | controller |       09:53:18 | watcher-cinder | Skipping creation of user-supplied namespace: watcher-kuttl-default\n
2026-03-14 09:56:26.619294 | controller |       \   logger.go:42: 09:53:18 | watcher-cinder/0-cleanup-watcher | starting test step
2026-03-14 09:56:26.619299 | controller |       0-cleanup-watcher\n    logger.go:42: 09:53:18 | watcher-cinder/0-cleanup-watcher
2026-03-14 09:56:26.619303 | controller |       | test step completed 0-cleanup-watcher\n    logger.go:42: 09:53:18 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619307 | controller |       | starting test step 1-deploy-watcher-no-cinder\n    logger.go:42: 09:53:18 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619312 | controller |       | Watcher:watcher-kuttl-default/watcher-kuttl created\n    logger.go:42: 09:53:18
2026-03-14 09:56:26.619316 | controller |       | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo
2026-03-14 09:56:26.619320 | controller |       pipefail\n        # check that the decision engine correctly detects that there
2026-03-14 09:56:26.619325 | controller |       is no cinder service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.619329 | controller |       | grep -c 'Block storage service is not enabled, skipping storage collector')\"
2026-03-14 09:56:26.619333 | controller |       == 2 ]\n        ]\n    logger.go:42: 09:53:18 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619338 | controller |       | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.619342 | controller |       09:53:18 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage
2026-03-14 09:56:26.619352 | controller |       service is not enabled, skipping storage collector'\n    logger.go:42: 09:53:18
2026-03-14 09:56:26.619356 | controller |       | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound):
2026-03-14 09:56:26.619361 | controller |       pods \"watcher-kuttl-decision-engine-0\" not found in namespace \"watcher-kuttl-default\"\n
2026-03-14 09:56:26.619365 | controller |       \   logger.go:42: 09:53:18 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0
2026-03-14 09:56:26.619369 | controller |       == 2 ']'\n    logger.go:42: 09:53:20 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619374 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.619378 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.619382 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:56:26.619386 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.619391 | controller |       09:53:20 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.619395 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:20 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619399 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.619403 | controller |       \   logger.go:42: 09:53:20 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:56:26.619409 | controller |       error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found
2026-03-14 09:56:26.619414 | controller |       in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:53:20 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619418 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:53:21 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619422 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.619427 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.619431 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:56:26.619435 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.619439 | controller |       09:53:21 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.619444 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:21 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619448 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.619457 | controller |       \   logger.go:42: 09:53:21 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:56:26.619462 | controller |       error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found
2026-03-14 09:56:26.619466 | controller |       in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:53:21 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619470 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:53:22 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619478 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.619483 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.619487 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:56:26.619491 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.619496 | controller |       09:53:22 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.619500 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:22 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619504 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.619509 | controller |       \   logger.go:42: 09:53:22 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:56:26.619513 | controller |       error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found
2026-03-14 09:56:26.619517 | controller |       in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:53:22 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619522 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:53:23 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619526 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.619530 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.619534 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:56:26.619539 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.619543 | controller |       09:53:23 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.619547 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:23 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619552 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.619556 | controller |       \   logger.go:42: 09:53:23 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:56:26.619563 | controller |       error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found
2026-03-14 09:56:26.619568 | controller |       in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:53:23 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619572 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:53:24 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619576 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.619581 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.619585 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:56:26.619589 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.619593 | controller |       09:53:24 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.619611 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:24 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619616 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.619620 | controller |       \   logger.go:42: 09:53:24 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:56:26.619624 | controller |       error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found
2026-03-14 09:56:26.619629 | controller |       in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:53:24 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619633 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:53:25 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619637 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.619642 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.619646 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:56:26.619650 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.619654 | controller |       09:53:25 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.619659 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:25 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619663 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.619673 | controller |       \   logger.go:42: 09:53:26 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:56:26.619678 | controller |       error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found
2026-03-14 09:56:26.619682 | controller |       in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:53:26 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619688 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:53:27 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619693 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.619697 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.619701 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:56:26.619706 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.619710 | controller |       09:53:27 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.619714 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:27 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619718 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.619723 | controller |       \   logger.go:42: 09:53:27 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:56:26.619727 | controller |       error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found
2026-03-14 09:56:26.619733 | controller |       in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:53:27 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619738 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:53:28 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619742 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.619746 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.619751 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:56:26.619755 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.619759 | controller |       09:53:28 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.619763 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:28 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619768 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.619772 | controller |       \   logger.go:42: 09:53:28 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:56:26.619776 | controller |       error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found
2026-03-14 09:56:26.619780 | controller |       in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:53:28 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619785 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:53:29 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619789 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.619793 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.619799 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:56:26.619803 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.619808 | controller |       09:53:29 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.619812 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:29 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619816 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.619821 | controller |       \   logger.go:42: 09:53:29 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:56:26.619825 | controller |       error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found
2026-03-14 09:56:26.619829 | controller |       in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:53:29 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619833 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:53:30 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619838 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.619842 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.619846 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:56:26.619850 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.619855 | controller |       09:53:30 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage
2026-03-14 09:56:26.619859 | controller |       service is not enabled, skipping storage collector'\n    logger.go:42: 09:53:30
2026-03-14 09:56:26.619863 | controller |       | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.619873 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:30 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619877 | controller |       | error: error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\"
2026-03-14 09:56:26.619883 | controller |       not found in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:53:30 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619888 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:53:31 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619892 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.619896 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.619901 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:56:26.619905 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.619909 | controller |       09:53:31 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.619913 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:31 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619918 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.619922 | controller |       \   logger.go:42: 09:53:31 | watcher-cinder/1-deploy-watcher-no-cinder | Error from
2026-03-14 09:56:26.619926 | controller |       server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\"
2026-03-14 09:56:26.619931 | controller |       is waiting to start: ContainerCreating\n    logger.go:42: 09:53:31 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619935 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:53:32 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619939 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.619944 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.619948 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:56:26.619952 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.619957 | controller |       09:53:32 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.619961 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:32 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619965 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.619970 | controller |       \   logger.go:42: 09:53:33 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0
2026-03-14 09:56:26.619974 | controller |       == 2 ']'\n    logger.go:42: 09:53:34 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.619978 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.619982 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.619987 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:56:26.619991 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.619995 | controller |       09:53:34 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.620000 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:34 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.620004 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.620010 | controller |       \   logger.go:42: 09:53:34 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0
2026-03-14 09:56:26.620014 | controller |       == 2 ']'\n    logger.go:42: 09:53:35 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.620019 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620023 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.620027 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:56:26.620033 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.620037 | controller |       09:53:35 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.620042 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:35 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.620046 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.620050 | controller |       \   logger.go:42: 09:53:35 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0
2026-03-14 09:56:26.620054 | controller |       == 2 ']'\n    logger.go:42: 09:53:36 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.620059 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620063 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.620073 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:56:26.620078 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.620082 | controller |       09:53:36 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.620086 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:36 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.620090 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.620095 | controller |       \   logger.go:42: 09:53:36 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2
2026-03-14 09:56:26.620099 | controller |       == 2 ']'\n    logger.go:42: 09:53:37 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.620103 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620107 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.620112 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:56:26.620116 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.620120 | controller |       09:53:37 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.620125 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:37 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.620129 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.620133 | controller |       \   logger.go:42: 09:53:37 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2
2026-03-14 09:56:26.620137 | controller |       == 2 ']'\n    logger.go:42: 09:53:38 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.620142 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620146 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.620150 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:56:26.620154 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.620159 | controller |       09:53:38 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.620163 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:38 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.620167 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.620172 | controller |       \   logger.go:42: 09:53:39 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2
2026-03-14 09:56:26.620176 | controller |       == 2 ']'\n    logger.go:42: 09:53:40 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.620180 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620188 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.620192 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:56:26.620196 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.620202 | controller |       09:53:40 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.620206 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:40 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.620211 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.620215 | controller |       \   logger.go:42: 09:53:40 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2
2026-03-14 09:56:26.620219 | controller |       == 2 ']'\n    logger.go:42: 09:53:41 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.620223 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620228 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.620232 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:56:26.620236 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.620240 | controller |       09:53:41 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.620245 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:41 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.620249 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.620253 | controller |       \   logger.go:42: 09:53:41 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2
2026-03-14 09:56:26.620257 | controller |       == 2 ']'\n    logger.go:42: 09:53:42 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.620264 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620270 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.620282 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:56:26.620289 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.620295 | controller |       09:53:42 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.620300 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:42 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.620305 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.620309 | controller |       \   logger.go:42: 09:53:42 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2
2026-03-14 09:56:26.620313 | controller |       == 2 ']'\n    logger.go:42: 09:53:42 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:56:26.620318 | controller |       | test step completed 1-deploy-watcher-no-cinder\n    logger.go:42: 09:53:42 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620322 | controller |       | starting test step 2-deploy-cinder\n    logger.go:42: 09:53:42 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620326 | controller |       | OpenStackControlPlane:watcher-kuttl-default/openstack updated\n    logger.go:42:
2026-03-14 09:56:26.620330 | controller |       09:53:42 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:56:26.620335 | controller |       \       # check that the decision detects that there is a cinder service and\n        #
2026-03-14 09:56:26.620339 | controller |       does not log that storage collector is skipped\n        [ \"$(oc logs -n $NAMESPACE
2026-03-14 09:56:26.620343 | controller |       watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled,
2026-03-14 09:56:26.620347 | controller |       skipping storage collector')\" == 0 ]\n        ]\n    logger.go:42: 09:53:42 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620354 | controller |       | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.620359 | controller |       09:53:42 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.620363 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:53:43 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620367 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:53:44 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620371 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620376 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.620380 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.620384 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.620391 | controller |       0 ]\n        ]\n    logger.go:42: 09:53:44 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.620395 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.620399 | controller |       09:53:44 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.620403 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:53:44 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620408 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:53:45 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620412 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620416 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.620420 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.620424 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.620429 | controller |       0 ]\n        ]\n    logger.go:42: 09:53:45 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.620433 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.620437 | controller |       09:53:45 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.620442 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:53:45 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620446 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:53:46 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620450 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620454 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.620459 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.620463 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.620467 | controller |       0 ]\n        ]\n    logger.go:42: 09:53:46 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.620471 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.620476 | controller |       09:53:46 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.620485 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:53:46 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620490 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:53:47 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620494 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620498 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.620503 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.620509 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.620513 | controller |       0 ]\n        ]\n    logger.go:42: 09:53:47 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.620518 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.620522 | controller |       09:53:47 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.620526 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:53:48 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620530 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:53:49 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620535 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620539 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.620543 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.620548 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.620552 | controller |       0 ]\n        ]\n    logger.go:42: 09:53:49 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.620556 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.620560 | controller |       09:53:49 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.620565 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:53:49 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620572 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:53:50 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620576 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620581 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.620585 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.620589 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.620605 | controller |       0 ]\n        ]\n    logger.go:42: 09:53:50 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.620612 | controller |       grep -c 'Block storage service is not enabled, skipping storage collector'\n    logger.go:42:
2026-03-14 09:56:26.620616 | controller |       09:53:50 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.620622 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:53:50 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620628 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:53:51 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620634 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620639 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.620643 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.620648 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.620652 | controller |       0 ]\n        ]\n    logger.go:42: 09:53:51 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.620656 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.620660 | controller |       09:53:51 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.620665 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:53:51 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620669 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:53:52 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620673 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620680 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.620684 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.620688 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.620693 | controller |       0 ]\n        ]\n    logger.go:42: 09:53:52 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.620697 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.620701 | controller |       09:53:52 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.620705 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:53:52 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620717 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:53:53 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620722 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620726 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.620730 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.620735 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.620739 | controller |       0 ]\n        ]\n    logger.go:42: 09:53:53 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.620743 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.620747 | controller |       09:53:53 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.620754 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:53:54 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620759 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:53:55 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620763 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620767 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.620771 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.620776 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.620782 | controller |       0 ]\n        ]\n    logger.go:42: 09:53:55 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.620786 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.620790 | controller |       09:53:55 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.620796 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:53:55 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620801 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:53:56 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620806 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620810 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.620814 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.620818 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.620823 | controller |       0 ]\n        ]\n    logger.go:42: 09:53:56 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.620827 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.620831 | controller |       09:53:56 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.620836 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:53:56 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620842 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:53:57 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620846 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620850 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.620854 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.620859 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.620863 | controller |       0 ]\n        ]\n    logger.go:42: 09:53:57 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.620867 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.620872 | controller |       09:53:57 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.620876 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:53:57 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620880 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:53:58 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620884 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620889 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.620893 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.620897 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.620901 | controller |       0 ]\n        ]\n    logger.go:42: 09:53:58 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.620906 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.620910 | controller |       09:53:58 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.620914 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:53:59 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620925 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:00 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620929 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620934 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.620938 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.620942 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.620947 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:00 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.620951 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.620955 | controller |       09:54:00 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.620959 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:00 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620965 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:01 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.620970 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.620974 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.620978 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.620983 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.620987 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:01 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.620991 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.620996 | controller |       09:54:01 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621002 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:01 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621006 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:02 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621010 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621015 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621019 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621023 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621028 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:02 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621032 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.621036 | controller |       09:54:02 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621040 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:02 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621045 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:03 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621049 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621053 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621058 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621062 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621066 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:03 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621070 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.621075 | controller |       09:54:03 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621079 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:04 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621083 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:05 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621088 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621092 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621096 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621100 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621105 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:05 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621109 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.621113 | controller |       09:54:05 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621118 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:05 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621128 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:06 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621133 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621137 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621141 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621147 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621152 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:06 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621158 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.621162 | controller |       09:54:06 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621166 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:06 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621171 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:07 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621175 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621179 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621184 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621188 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621192 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:07 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621196 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.621201 | controller |       09:54:07 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621205 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:07 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621209 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:08 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621214 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621218 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621222 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621227 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621231 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:08 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621235 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.621239 | controller |       09:54:08 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621244 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:09 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621248 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:10 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621252 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621257 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621261 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621265 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621270 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:10 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621274 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.621278 | controller |       09:54:10 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621282 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:10 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621287 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:11 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621291 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621295 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621300 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621305 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621310 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:11 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621314 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.621320 | controller |       09:54:11 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621324 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:11 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621336 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:12 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621342 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621348 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621356 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621363 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621369 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:12 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621375 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.621381 | controller |       09:54:12 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621388 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:12 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621392 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:13 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621396 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621401 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621405 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621409 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621414 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:13 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621418 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.621422 | controller |       09:54:13 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621426 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:14 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621431 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:15 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621435 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621439 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621444 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621448 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621452 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:15 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621457 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.621461 | controller |       09:54:15 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621465 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:15 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621469 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:16 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621474 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621478 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621484 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621488 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621493 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:16 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621497 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.621501 | controller |       09:54:16 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621506 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:16 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621510 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:17 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621514 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621520 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621524 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621529 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621533 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:17 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621537 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.621541 | controller |       09:54:17 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621546 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:17 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621550 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:18 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621561 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621566 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621570 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621575 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621579 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:18 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621583 | controller |       grep -c 'Block storage service is not enabled, skipping storage collector'\n    logger.go:42:
2026-03-14 09:56:26.621588 | controller |       09:54:18 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.621592 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:54:18 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621610 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:19 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621615 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621619 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621623 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621628 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621632 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:19 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621636 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.621640 | controller |       09:54:19 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621645 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:20 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621649 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:21 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621657 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621661 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621665 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621670 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621674 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:21 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621678 | controller |       grep -c 'Block storage service is not enabled, skipping storage collector'\n    logger.go:42:
2026-03-14 09:56:26.621683 | controller |       09:54:21 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.621687 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:54:21 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621691 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:22 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621695 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621700 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621704 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621708 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621712 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:22 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621717 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.621721 | controller |       09:54:22 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621725 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:22 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621731 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:23 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621736 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621740 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621744 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621748 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621753 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:23 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621757 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.621761 | controller |       09:54:23 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621765 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:23 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621770 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:24 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621780 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621785 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621789 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621794 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621798 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:24 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621802 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.621807 | controller |       09:54:24 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621813 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:25 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621817 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:26 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621822 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621826 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621830 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621835 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621839 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:26 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621843 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.621848 | controller |       09:54:26 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621852 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:26 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621856 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:27 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621860 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621865 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621869 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621873 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621877 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:27 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621882 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.621886 | controller |       09:54:27 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621890 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:27 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621895 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:28 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621899 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621903 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621907 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621913 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621918 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:28 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621923 | controller |       grep -c 'Block storage service is not enabled, skipping storage collector'\n    logger.go:42:
2026-03-14 09:56:26.621928 | controller |       09:54:28 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.621932 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:54:28 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621936 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:29 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621941 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621945 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621949 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.621953 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.621958 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:29 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.621964 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.621968 | controller |       09:54:29 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.621972 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:29 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621976 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:30 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.621987 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.621991 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.621996 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.622000 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.622004 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:30 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.622009 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.622013 | controller |       09:54:30 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.622017 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:31 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622021 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:32 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622026 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.622030 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.622034 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.622039 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.622043 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:32 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.622047 | controller |       grep -c 'Block storage service is not enabled, skipping storage collector'\n    logger.go:42:
2026-03-14 09:56:26.622052 | controller |       09:54:32 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.622056 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:54:32 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622060 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:33 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622064 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.622069 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.622073 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.622077 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.622082 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:33 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.622086 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.622090 | controller |       09:54:33 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.622094 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:33 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622099 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:34 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622105 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.622110 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.622115 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.622119 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.622125 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:34 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.622129 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.622134 | controller |       09:54:34 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.622138 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:34 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622142 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:35 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622147 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.622151 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.622155 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.622160 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.622164 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:35 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.622168 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.622172 | controller |       09:54:35 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.622177 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:36 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622181 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:37 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622185 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.622196 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.622200 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.622205 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.622209 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:37 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.622213 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.622218 | controller |       09:54:37 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.622222 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:37 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622226 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:38 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622231 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.622235 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.622239 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.622243 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.622248 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:38 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.622252 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.622256 | controller |       09:54:38 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.622261 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:38 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622265 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:39 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622269 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.622274 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.622279 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.622284 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.622290 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:39 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.622294 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.622298 | controller |       09:54:39 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.622303 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:39 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622307 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:40 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622311 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.622316 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.622320 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.622324 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.622328 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:40 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.622333 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.622337 | controller |       09:54:40 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.622341 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:41 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622346 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:54:42 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622350 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.622354 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.622360 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.622364 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.622369 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:42 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.622373 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.622377 | controller |       09:54:42 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.622381 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:42 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622386 | controller |       | Error from server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\"
2026-03-14 09:56:26.622396 | controller |       is waiting to start: ContainerCreating\n    logger.go:42: 09:54:42 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622401 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:54:43 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622405 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.622409 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.622414 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.622418 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.622422 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:43 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.622428 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.622434 | controller |       09:54:43 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.622443 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:43 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622449 | controller |       | Error from server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\"
2026-03-14 09:56:26.622458 | controller |       is waiting to start: ContainerCreating\n    logger.go:42: 09:54:43 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622463 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:54:44 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622467 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.622472 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.622476 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.622480 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.622484 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:44 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.622489 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.622493 | controller |       09:54:44 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.622497 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:44 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622503 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:54:45 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622507 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.622512 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.622516 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.622520 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.622524 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:45 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.622529 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.622533 | controller |       09:54:45 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.622537 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:45 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622541 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:54:46 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622546 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.622550 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.622554 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.622558 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.622563 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:46 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.622567 | controller |       grep -c 'Block storage service is not enabled, skipping storage collector'\n    logger.go:42:
2026-03-14 09:56:26.622571 | controller |       09:54:46 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.622575 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:54:47 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622580 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:54:48 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622584 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.622588 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.622592 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.622612 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.622617 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:48 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.622628 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.622633 | controller |       09:54:48 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.622638 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:48 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622642 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:54:49 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622647 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.622652 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.622659 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.622665 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.622671 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:49 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.622676 | controller |       grep -c 'Block storage service is not enabled, skipping storage collector'\n    logger.go:42:
2026-03-14 09:56:26.622682 | controller |       09:54:49 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.622687 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:54:49 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622691 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:54:50 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622695 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.622699 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.622704 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.622708 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.622712 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:50 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.622721 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.622725 | controller |       09:54:50 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.622729 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:50 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622734 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:54:51 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622738 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.622742 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.622746 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.622751 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.622755 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:51 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.622759 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.622764 | controller |       09:54:51 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.622768 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:52 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622772 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:54:53 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622776 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.622783 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.622787 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.622791 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.622796 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:53 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.622800 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.622804 | controller |       09:54:53 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.622808 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:53 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622812 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:54:54 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622817 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.622821 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:56:26.622825 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.622829 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.622834 | controller |       0 ]\n        ]\n    logger.go:42: 09:54:54 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:56:26.622838 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.622850 | controller |       09:54:54 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.622854 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:54 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622859 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:54:54 | watcher-cinder/2-deploy-cinder
2026-03-14 09:56:26.622863 | controller |       | test step completed 2-deploy-cinder\n    logger.go:42: 09:54:54 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.622867 | controller |       | starting test step 3-remove-cinder\n    logger.go:42: 09:54:54 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.622871 | controller |       | OpenStackControlPlane:watcher-kuttl-default/openstack updated\n    logger.go:42:
2026-03-14 09:56:26.622876 | controller |       09:54:54 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:56:26.622880 | controller |       \       # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.622884 | controller |       service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep
2026-03-14 09:56:26.622889 | controller |       -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n
2026-03-14 09:56:26.622893 | controller |       \       ]\n    logger.go:42: 09:54:54 | watcher-cinder/3-remove-cinder | ++ oc logs
2026-03-14 09:56:26.622897 | controller |       -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42: 09:54:54
2026-03-14 09:56:26.622903 | controller |       | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled,
2026-03-14 09:56:26.622908 | controller |       skipping storage collector'\n    logger.go:42: 09:54:54 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.622912 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:54:55 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.622916 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.622921 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.622925 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is
2026-03-14 09:56:26.622929 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.622933 | controller |       09:54:55 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.622953 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:55 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.622959 | controller |       | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.622964 | controller |       09:54:55 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n    logger.go:42:
2026-03-14 09:56:26.622969 | controller |       09:54:57 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:56:26.622973 | controller |       \       # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.622977 | controller |       service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep
2026-03-14 09:56:26.622982 | controller |       -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n
2026-03-14 09:56:26.622986 | controller |       \       ]\n    logger.go:42: 09:54:57 | watcher-cinder/3-remove-cinder | ++ oc logs
2026-03-14 09:56:26.622990 | controller |       -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42: 09:54:57
2026-03-14 09:56:26.622994 | controller |       | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled,
2026-03-14 09:56:26.622999 | controller |       skipping storage collector'\n    logger.go:42: 09:54:57 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623003 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:54:58 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623007 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.623011 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.623016 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is
2026-03-14 09:56:26.623020 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.623024 | controller |       09:54:58 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.623028 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:54:58 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623033 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.623037 | controller |       \   logger.go:42: 09:54:58 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n
2026-03-14 09:56:26.623041 | controller |       \   logger.go:42: 09:54:59 | watcher-cinder/3-remove-cinder | running command: [sh
2026-03-14 09:56:26.623045 | controller |       -c set -euxo pipefail\n        # check that the decision engine correctly detects
2026-03-14 09:56:26.623050 | controller |       that there is no cinder service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.623054 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.623058 | controller |       2 ]\n        ]\n    logger.go:42: 09:54:59 | watcher-cinder/3-remove-cinder | ++
2026-03-14 09:56:26.623072 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.623076 | controller |       09:54:59 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.623081 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:54:59 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623085 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:55:00 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623089 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.623094 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.623098 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is
2026-03-14 09:56:26.623102 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.623106 | controller |       09:55:00 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.623111 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:55:00 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623117 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.623121 | controller |       \   logger.go:42: 09:55:00 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n
2026-03-14 09:56:26.623125 | controller |       \   logger.go:42: 09:55:01 | watcher-cinder/3-remove-cinder | running command: [sh
2026-03-14 09:56:26.623130 | controller |       -c set -euxo pipefail\n        # check that the decision engine correctly detects
2026-03-14 09:56:26.623136 | controller |       that there is no cinder service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.623140 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.623144 | controller |       2 ]\n        ]\n    logger.go:42: 09:55:01 | watcher-cinder/3-remove-cinder | ++
2026-03-14 09:56:26.623148 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.623153 | controller |       09:55:01 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.623157 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:55:02 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623161 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:55:03 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623165 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.623170 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.623174 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is
2026-03-14 09:56:26.623178 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.623182 | controller |       09:55:03 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.623187 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:55:03 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623191 | controller |       | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.623195 | controller |       09:55:03 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n    logger.go:42:
2026-03-14 09:56:26.623199 | controller |       09:55:04 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:56:26.623204 | controller |       \       # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.623208 | controller |       service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep
2026-03-14 09:56:26.623212 | controller |       -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n
2026-03-14 09:56:26.623216 | controller |       \       ]\n    logger.go:42: 09:55:04 | watcher-cinder/3-remove-cinder | ++ oc logs
2026-03-14 09:56:26.623221 | controller |       -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42: 09:55:04
2026-03-14 09:56:26.623225 | controller |       | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled,
2026-03-14 09:56:26.623229 | controller |       skipping storage collector'\n    logger.go:42: 09:55:04 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623235 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:55:05 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623239 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.623244 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.623248 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is
2026-03-14 09:56:26.623252 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.623256 | controller |       09:55:05 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.623261 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:55:05 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623267 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.623277 | controller |       \   logger.go:42: 09:55:05 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n
2026-03-14 09:56:26.623282 | controller |       \   logger.go:42: 09:55:06 | watcher-cinder/3-remove-cinder | running command: [sh
2026-03-14 09:56:26.623286 | controller |       -c set -euxo pipefail\n        # check that the decision engine correctly detects
2026-03-14 09:56:26.623290 | controller |       that there is no cinder service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.623295 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.623299 | controller |       2 ]\n        ]\n    logger.go:42: 09:55:06 | watcher-cinder/3-remove-cinder | ++
2026-03-14 09:56:26.623303 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.623307 | controller |       09:55:06 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.623312 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:55:07 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623316 | controller |       | Error from server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\"
2026-03-14 09:56:26.623320 | controller |       is waiting to start: ContainerCreating\n    logger.go:42: 09:55:07 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623324 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:55:08 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623329 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.623333 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.623337 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is
2026-03-14 09:56:26.623342 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.623346 | controller |       09:55:08 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.623350 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:55:08 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623354 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.623359 | controller |       \   logger.go:42: 09:55:08 | watcher-cinder/3-remove-cinder | Error from server
2026-03-14 09:56:26.623363 | controller |       (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\"
2026-03-14 09:56:26.623367 | controller |       is waiting to start: ContainerCreating\n    logger.go:42: 09:55:08 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623371 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:55:09 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623376 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.623380 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.623384 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is
2026-03-14 09:56:26.623388 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.623393 | controller |       09:55:09 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.623397 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:55:09 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623401 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.623405 | controller |       \   logger.go:42: 09:55:09 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n
2026-03-14 09:56:26.623410 | controller |       \   logger.go:42: 09:55:10 | watcher-cinder/3-remove-cinder | running command: [sh
2026-03-14 09:56:26.623415 | controller |       -c set -euxo pipefail\n        # check that the decision engine correctly detects
2026-03-14 09:56:26.623421 | controller |       that there is no cinder service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.623426 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.623430 | controller |       2 ]\n        ]\n    logger.go:42: 09:55:10 | watcher-cinder/3-remove-cinder | ++
2026-03-14 09:56:26.623434 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.623439 | controller |       09:55:10 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.623443 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:55:10 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623447 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:55:11 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623451 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.623456 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.623460 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is
2026-03-14 09:56:26.623464 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.623469 | controller |       09:55:11 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.623479 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:55:11 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623483 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.623488 | controller |       \   logger.go:42: 09:55:11 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n
2026-03-14 09:56:26.623492 | controller |       \   logger.go:42: 09:55:12 | watcher-cinder/3-remove-cinder | running command: [sh
2026-03-14 09:56:26.623496 | controller |       -c set -euxo pipefail\n        # check that the decision engine correctly detects
2026-03-14 09:56:26.623501 | controller |       that there is no cinder service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.623506 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.623512 | controller |       2 ]\n        ]\n    logger.go:42: 09:55:12 | watcher-cinder/3-remove-cinder | ++
2026-03-14 09:56:26.623518 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.623524 | controller |       09:55:12 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.623529 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:55:12 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623535 | controller |       | + '[' 2 == 2 ']'\n    logger.go:42: 09:55:14 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623540 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.623546 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.623551 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is
2026-03-14 09:56:26.623556 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.623561 | controller |       09:55:14 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.623567 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:55:14 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623572 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:56:26.623578 | controller |       \   logger.go:42: 09:55:14 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n
2026-03-14 09:56:26.623583 | controller |       \   logger.go:42: 09:55:15 | watcher-cinder/3-remove-cinder | running command: [sh
2026-03-14 09:56:26.623589 | controller |       -c set -euxo pipefail\n        # check that the decision engine correctly detects
2026-03-14 09:56:26.623611 | controller |       that there is no cinder service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.623620 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:56:26.623626 | controller |       2 ]\n        ]\n    logger.go:42: 09:55:15 | watcher-cinder/3-remove-cinder | ++
2026-03-14 09:56:26.623635 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.623641 | controller |       09:55:15 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.623647 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:55:15 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623653 | controller |       | + '[' 2 == 2 ']'\n    logger.go:42: 09:55:16 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623659 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.623665 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.623671 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is
2026-03-14 09:56:26.623677 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.623683 | controller |       09:55:16 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.623689 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:55:16 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623695 | controller |       | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.623701 | controller |       09:55:16 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n    logger.go:42:
2026-03-14 09:56:26.623707 | controller |       09:55:17 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:56:26.623714 | controller |       \       # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.623720 | controller |       service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep
2026-03-14 09:56:26.623726 | controller |       -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n
2026-03-14 09:56:26.623733 | controller |       \       ]\n    logger.go:42: 09:55:17 | watcher-cinder/3-remove-cinder | ++ oc logs
2026-03-14 09:56:26.623739 | controller |       -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42: 09:55:17
2026-03-14 09:56:26.623745 | controller |       | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled,
2026-03-14 09:56:26.623751 | controller |       skipping storage collector'\n    logger.go:42: 09:55:17 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623769 | controller |       | + '[' 2 == 2 ']'\n    logger.go:42: 09:55:18 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623776 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:56:26.623783 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:56:26.623789 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is
2026-03-14 09:56:26.623795 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:56:26.623801 | controller |       09:55:18 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:56:26.623807 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:55:18 | watcher-cinder/3-remove-cinder
2026-03-14 09:56:26.623814 | controller |       | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:56:26.623820 | controller |       09:55:19 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n    logger.go:42:
2026-03-14 09:56:26.623826 | controller |       09:55:19 | watcher-cinder/3-remove-cinder | test step completed 3-remove-cinder\n
2026-03-14 09:56:26.623832 | controller |       \   logger.go:42: 09:55:19 | watcher-cinder/4-cleanup-watcher | starting test step
2026-03-14 09:56:26.623842 | controller |       4-cleanup-watcher\n    logger.go:42: 09:55:26 | watcher-cinder/4-cleanup-watcher
2026-03-14 09:56:26.623848 | controller |       | test step completed 4-cleanup-watcher\n    logger.go:42: 09:55:26 | watcher-cinder
2026-03-14 09:56:26.623854 | controller |       | skipping kubernetes event logging\n=== CONT  kuttl/harness/watcher-api-scaling\n
2026-03-14 09:56:26.623863 | controller |       \   logger.go:42: 09:55:26 | watcher-api-scaling | Skipping creation of user-supplied
2026-03-14 09:56:26.623869 | controller |       namespace: watcher-kuttl-default\n    logger.go:42: 09:55:26 | watcher-api-scaling/0-cleanup-watcher
2026-03-14 09:56:26.623875 | controller |       | starting test step 0-cleanup-watcher\n    logger.go:42: 09:55:26 | watcher-api-scaling/0-cleanup-watcher
2026-03-14 09:56:26.623881 | controller |       | test step completed 0-cleanup-watcher\n    logger.go:42: 09:55:26 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.623888 | controller |       | starting test step 1-deploy-with-defaults\n    logger.go:42: 09:55:26 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.623894 | controller |       | Watcher:watcher-kuttl-default/watcher-kuttl created\n    logger.go:42: 09:55:27
2026-03-14 09:56:26.623900 | controller |       | watcher-api-scaling/1-deploy-with-defaults | running command: [sh -c set -euxo
2026-03-14 09:56:26.623907 | controller |       pipefail\n        oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:56:26.623913 | controller |       service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc
2026-03-14 09:56:26.623919 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.623925 | controller |       -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n        [ $(oc get -n watcher-kuttl-default
2026-03-14 09:56:26.623932 | controller |       keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [
2026-03-14 09:56:26.623938 | controller |       -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\"
2026-03-14 09:56:26.623944 | controller |       ]\n        # If we are running the container locally, skip following test\n        if
2026-03-14 09:56:26.623951 | controller |       [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:56:26.623957 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:56:26.623964 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:56:26.623969 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:56:26.623976 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:56:26.623982 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:56:26.623988 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:56:26.623995 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:56:26.624001 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:55:27 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624008 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.624019 | controller |       value -c Name -c Type\n    logger.go:42: 09:55:27 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624024 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:55:29 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624030 | controller |       | + '[' 0 == 1 ']'\n    logger.go:42: 09:55:30 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624036 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.624041 | controller |       openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c
2026-03-14 09:56:26.624057 | controller |       ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.624066 | controller |       -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print
2026-03-14 09:56:26.624071 | controller |       $1}')\n        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.624077 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl
2026-03-14 09:56:26.624082 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # If we are running the container
2026-03-14 09:56:26.624088 | controller |       locally, skip following test\n        if [ \"$(oc get pods -n openstack-operators
2026-03-14 09:56:26.624093 | controller |       -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit
2026-03-14 09:56:26.624099 | controller |       0\n        fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:56:26.624104 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:56:26.624110 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:56:26.624115 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:56:26.624120 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:56:26.624126 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:56:26.624131 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:56:26.624137 | controller |       \       ]\n    logger.go:42: 09:55:30 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624142 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.624148 | controller |       value -c Name -c Type\n    logger.go:42: 09:55:30 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624154 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:55:32 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624160 | controller |       | + '[' 0 == 1 ']'\n    logger.go:42: 09:55:33 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624166 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.624172 | controller |       openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c
2026-03-14 09:56:26.624178 | controller |       ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.624184 | controller |       -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print
2026-03-14 09:56:26.624190 | controller |       $1}')\n        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.624196 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl
2026-03-14 09:56:26.624206 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # If we are running the container
2026-03-14 09:56:26.624212 | controller |       locally, skip following test\n        if [ \"$(oc get pods -n openstack-operators
2026-03-14 09:56:26.624219 | controller |       -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit
2026-03-14 09:56:26.624225 | controller |       0\n        fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:56:26.624231 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:56:26.624237 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:56:26.624243 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:56:26.624250 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:56:26.624256 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:56:26.624265 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:56:26.624271 | controller |       \       ]\n    logger.go:42: 09:55:33 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624278 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.624284 | controller |       value -c Name -c Type\n    logger.go:42: 09:55:33 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624290 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:55:36 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624296 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:55:36 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624302 | controller |       | ++ grep watcher\n    logger.go:42: 09:55:36 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624309 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:55:36 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624315 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.624332 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:55:38 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624340 | controller |       | + SERVICEID=895be62000ea4e65989b573b132a3aa1\n    logger.go:42: 09:55:38 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624346 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.624353 | controller |       \   logger.go:42: 09:55:38 | watcher-api-scaling/1-deploy-with-defaults | + '['
2026-03-14 09:56:26.624359 | controller |       895be62000ea4e65989b573b132a3aa1 == 895be62000ea4e65989b573b132a3aa1 ']'\n    logger.go:42:
2026-03-14 09:56:26.624365 | controller |       09:55:38 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.624371 | controller |       watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n    logger.go:42: 09:55:38
2026-03-14 09:56:26.624378 | controller |       | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.624384 | controller |       ']'\n    logger.go:42: 09:55:38 | watcher-api-scaling/1-deploy-with-defaults | ++
2026-03-14 09:56:26.624390 | controller |       oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n
2026-03-14 09:56:26.624396 | controller |       \   logger.go:42: 09:55:38 | watcher-api-scaling/1-deploy-with-defaults | + '['
2026-03-14 09:56:26.624403 | controller |       '' == '' ']'\n    logger.go:42: 09:55:38 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624409 | controller |       | + exit 0\n    logger.go:42: 09:55:39 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624415 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.624421 | controller |       openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c
2026-03-14 09:56:26.624428 | controller |       ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.624434 | controller |       -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print
2026-03-14 09:56:26.624440 | controller |       $1}')\n        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.624446 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl
2026-03-14 09:56:26.624452 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # If we are running the container
2026-03-14 09:56:26.624458 | controller |       locally, skip following test\n        if [ \"$(oc get pods -n openstack-operators
2026-03-14 09:56:26.624465 | controller |       -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit
2026-03-14 09:56:26.624471 | controller |       0\n        fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:56:26.624477 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:56:26.624487 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:56:26.624492 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:56:26.624498 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:56:26.624503 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:56:26.624509 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:56:26.624514 | controller |       \       ]\n    logger.go:42: 09:55:39 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624523 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.624528 | controller |       value -c Name -c Type\n    logger.go:42: 09:55:39 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624534 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:55:41 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624539 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:55:41 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624545 | controller |       | ++ grep watcher\n    logger.go:42: 09:55:41 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624553 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:55:41 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624558 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.624564 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:55:43 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624569 | controller |       | + SERVICEID=895be62000ea4e65989b573b132a3aa1\n    logger.go:42: 09:55:43 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624575 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.624580 | controller |       \   logger.go:42: 09:55:43 | watcher-api-scaling/1-deploy-with-defaults | + '['
2026-03-14 09:56:26.624586 | controller |       895be62000ea4e65989b573b132a3aa1 == 895be62000ea4e65989b573b132a3aa1 ']'\n    logger.go:42:
2026-03-14 09:56:26.624615 | controller |       09:55:43 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.624625 | controller |       watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n    logger.go:42: 09:55:43
2026-03-14 09:56:26.624631 | controller |       | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.624637 | controller |       ']'\n    logger.go:42: 09:55:43 | watcher-api-scaling/1-deploy-with-defaults | ++
2026-03-14 09:56:26.624642 | controller |       oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n
2026-03-14 09:56:26.624647 | controller |       \   logger.go:42: 09:55:44 | watcher-api-scaling/1-deploy-with-defaults | + '['
2026-03-14 09:56:26.624653 | controller |       '' == '' ']'\n    logger.go:42: 09:55:44 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624658 | controller |       | + exit 0\n    logger.go:42: 09:55:45 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624664 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.624669 | controller |       openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c
2026-03-14 09:56:26.624675 | controller |       ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.624680 | controller |       -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print
2026-03-14 09:56:26.624686 | controller |       $1}')\n        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.624691 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl
2026-03-14 09:56:26.624700 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # If we are running the container
2026-03-14 09:56:26.624705 | controller |       locally, skip following test\n        if [ \"$(oc get pods -n openstack-operators
2026-03-14 09:56:26.624711 | controller |       -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit
2026-03-14 09:56:26.624716 | controller |       0\n        fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:56:26.624722 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:56:26.624728 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:56:26.624734 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:56:26.624740 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:56:26.624746 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:56:26.624752 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:56:26.624758 | controller |       \       ]\n    logger.go:42: 09:55:45 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624764 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.624770 | controller |       value -c Name -c Type\n    logger.go:42: 09:55:45 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624776 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:55:47 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624783 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:55:47 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624789 | controller |       | ++ grep watcher\n    logger.go:42: 09:55:47 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624795 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:55:47 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624802 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.624808 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:55:49 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624814 | controller |       | + SERVICEID=895be62000ea4e65989b573b132a3aa1\n    logger.go:42: 09:55:49 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624819 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.624825 | controller |       \   logger.go:42: 09:55:49 | watcher-api-scaling/1-deploy-with-defaults | + '['
2026-03-14 09:56:26.624835 | controller |       895be62000ea4e65989b573b132a3aa1 == 895be62000ea4e65989b573b132a3aa1 ']'\n    logger.go:42:
2026-03-14 09:56:26.624841 | controller |       09:55:49 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.624847 | controller |       watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n    logger.go:42: 09:55:49
2026-03-14 09:56:26.624853 | controller |       | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.624859 | controller |       ']'\n    logger.go:42: 09:55:49 | watcher-api-scaling/1-deploy-with-defaults | ++
2026-03-14 09:56:26.624865 | controller |       oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n
2026-03-14 09:56:26.624881 | controller |       \   logger.go:42: 09:55:49 | watcher-api-scaling/1-deploy-with-defaults | + '['
2026-03-14 09:56:26.624887 | controller |       '' == '' ']'\n    logger.go:42: 09:55:49 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624893 | controller |       | + exit 0\n    logger.go:42: 09:55:50 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.624898 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.624909 | controller |       openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c
2026-03-14 09:56:26.624915 | controller |       ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.624921 | controller |       -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print
2026-03-14 09:56:26.624927 | controller |       $1}')\n        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.624933 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl
2026-03-14 09:56:26.624938 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # If we are running the container
2026-03-14 09:56:26.624944 | controller |       locally, skip following test\n        if [ \"$(oc get pods -n openstack-operators
2026-03-14 09:56:26.624950 | controller |       -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit
2026-03-14 09:56:26.624956 | controller |       0\n        fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:56:26.624962 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:56:26.624968 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:56:26.624974 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:56:26.624979 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:56:26.624985 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:56:26.624991 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:56:26.624996 | controller |       \       ]\n    logger.go:42: 09:55:50 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.625002 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.625008 | controller |       value -c Name -c Type\n    logger.go:42: 09:55:50 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.625013 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:55:53 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.625019 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:55:53 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.625025 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.625030 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:55:53 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.625036 | controller |       | ++ grep watcher\n    logger.go:42: 09:55:53 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.625042 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:55:55 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.625047 | controller |       | + SERVICEID=895be62000ea4e65989b573b132a3aa1\n    logger.go:42: 09:55:55 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.625054 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.625059 | controller |       \   logger.go:42: 09:55:56 | watcher-api-scaling/1-deploy-with-defaults | + '['
2026-03-14 09:56:26.625065 | controller |       895be62000ea4e65989b573b132a3aa1 == 895be62000ea4e65989b573b132a3aa1 ']'\n    logger.go:42:
2026-03-14 09:56:26.625074 | controller |       09:55:56 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.625080 | controller |       watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n    logger.go:42: 09:55:56
2026-03-14 09:56:26.625086 | controller |       | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.625093 | controller |       ']'\n    logger.go:42: 09:55:56 | watcher-api-scaling/1-deploy-with-defaults | ++
2026-03-14 09:56:26.625102 | controller |       oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n
2026-03-14 09:56:26.625109 | controller |       \   logger.go:42: 09:55:56 | watcher-api-scaling/1-deploy-with-defaults | + '['
2026-03-14 09:56:26.625115 | controller |       '' == '' ']'\n    logger.go:42: 09:55:56 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.625121 | controller |       | + exit 0\n    logger.go:42: 09:55:57 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.625127 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.625133 | controller |       openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c
2026-03-14 09:56:26.625148 | controller |       ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.625155 | controller |       -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print
2026-03-14 09:56:26.625161 | controller |       $1}')\n        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.625167 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl
2026-03-14 09:56:26.625173 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # If we are running the container
2026-03-14 09:56:26.625179 | controller |       locally, skip following test\n        if [ \"$(oc get pods -n openstack-operators
2026-03-14 09:56:26.625184 | controller |       -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit
2026-03-14 09:56:26.625189 | controller |       0\n        fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:56:26.625195 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:56:26.625203 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:56:26.625208 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:56:26.625214 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:56:26.625219 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:56:26.625224 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:56:26.625229 | controller |       \       ]\n    logger.go:42: 09:55:57 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.625235 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.625241 | controller |       value -c Name -c Type\n    logger.go:42: 09:55:57 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.625246 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:55:59 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.625252 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:55:59 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.625258 | controller |       | ++ grep watcher\n    logger.go:42: 09:55:59 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.625264 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:55:59 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.625270 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.625278 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:56:01 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.625284 | controller |       | + SERVICEID=895be62000ea4e65989b573b132a3aa1\n    logger.go:42: 09:56:01 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.625290 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:56:26.625296 | controller |       \   logger.go:42: 09:56:02 | watcher-api-scaling/1-deploy-with-defaults | + '['
2026-03-14 09:56:26.625301 | controller |       895be62000ea4e65989b573b132a3aa1 == 895be62000ea4e65989b573b132a3aa1 ']'\n    logger.go:42:
2026-03-14 09:56:26.625311 | controller |       09:56:02 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.625317 | controller |       watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n    logger.go:42: 09:56:02
2026-03-14 09:56:26.625323 | controller |       | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.625329 | controller |       ']'\n    logger.go:42: 09:56:02 | watcher-api-scaling/1-deploy-with-defaults | ++
2026-03-14 09:56:26.625334 | controller |       oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n
2026-03-14 09:56:26.625340 | controller |       \   logger.go:42: 09:56:02 | watcher-api-scaling/1-deploy-with-defaults | + '['
2026-03-14 09:56:26.625346 | controller |       '' == '' ']'\n    logger.go:42: 09:56:02 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.625352 | controller |       | + exit 0\n    logger.go:42: 09:56:02 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:56:26.625358 | controller |       | test step completed 1-deploy-with-defaults\n    logger.go:42: 09:56:02 | watcher-api-scaling/2-scale-up-watcher-api
2026-03-14 09:56:26.625364 | controller |       | starting test step 2-scale-up-watcher-api\n    logger.go:42: 09:56:02 | watcher-api-scaling/2-scale-up-watcher-api
2026-03-14 09:56:26.625370 | controller |       | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json'
2026-03-14 09:56:26.625376 | controller |       -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/replicas\", \"value\":3}]'\n
2026-03-14 09:56:26.625381 | controller |       \       ]\n    logger.go:42: 09:56:02 | watcher-api-scaling/2-scale-up-watcher-api
2026-03-14 09:56:26.625387 | controller |       | watcher.watcher.openstack.org/watcher-kuttl patched\n    logger.go:42: 09:56:13
2026-03-14 09:56:26.625393 | controller |       | watcher-api-scaling/2-scale-up-watcher-api | test step completed 2-scale-up-watcher-api\n
2026-03-14 09:56:26.625407 | controller |       \   logger.go:42: 09:56:13 | watcher-api-scaling/3-scale-down-watcher-api | starting
2026-03-14 09:56:26.625413 | controller |       test step 3-scale-down-watcher-api\n    logger.go:42: 09:56:13 | watcher-api-scaling/3-scale-down-watcher-api
2026-03-14 09:56:26.625419 | controller |       | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json'
2026-03-14 09:56:26.625425 | controller |       -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/replicas\", \"value\":1}]'\n
2026-03-14 09:56:26.625431 | controller |       \       ]\n    logger.go:42: 09:56:14 | watcher-api-scaling/3-scale-down-watcher-api
2026-03-14 09:56:26.625437 | controller |       | watcher.watcher.openstack.org/watcher-kuttl patched\n    logger.go:42: 09:56:18
2026-03-14 09:56:26.625443 | controller |       | watcher-api-scaling/3-scale-down-watcher-api | test step completed 3-scale-down-watcher-api\n
2026-03-14 09:56:26.625449 | controller |       \   logger.go:42: 09:56:18 | watcher-api-scaling/4-scale-down-zero-watcher-api |
2026-03-14 09:56:26.625455 | controller |       starting test step 4-scale-down-zero-watcher-api\n    logger.go:42: 09:56:18 | watcher-api-scaling/4-scale-down-zero-watcher-api
2026-03-14 09:56:26.625460 | controller |       | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json'
2026-03-14 09:56:26.625466 | controller |       -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/replicas\", \"value\":0}]'\n
2026-03-14 09:56:26.625472 | controller |       \       ]\n    logger.go:42: 09:56:18 | watcher-api-scaling/4-scale-down-zero-watcher-api
2026-03-14 09:56:26.625478 | controller |       | watcher.watcher.openstack.org/watcher-kuttl patched\n    logger.go:42: 09:56:19
2026-03-14 09:56:26.625484 | controller |       | watcher-api-scaling/4-scale-down-zero-watcher-api | test step completed 4-scale-down-zero-watcher-api\n
2026-03-14 09:56:26.625490 | controller |       \   logger.go:42: 09:56:19 | watcher-api-scaling/5-cleanup-watcher | starting test
2026-03-14 09:56:26.625495 | controller |       step 5-cleanup-watcher\n    logger.go:42: 09:56:19 | watcher-api-scaling/5-cleanup-watcher
2026-03-14 09:56:26.625501 | controller |       | running command: [sh -c set -ex\n        oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.625510 | controller |       -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0
2026-03-14 09:56:26.625516 | controller |       ]\n        ]\n    logger.go:42: 09:56:19 | watcher-api-scaling/5-cleanup-watcher
2026-03-14 09:56:26.625521 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.625527 | controller |       value -c Name -c Type\n    logger.go:42: 09:56:19 | watcher-api-scaling/5-cleanup-watcher
2026-03-14 09:56:26.625533 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:56:22 | watcher-api-scaling/5-cleanup-watcher
2026-03-14 09:56:26.625539 | controller |       | + '[' 1 == 0 ']'\n    logger.go:42: 09:56:23 | watcher-api-scaling/5-cleanup-watcher
2026-03-14 09:56:26.625544 | controller |       | running command: [sh -c set -ex\n        oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:56:26.625550 | controller |       -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0
2026-03-14 09:56:26.625556 | controller |       ]\n        ]\n    logger.go:42: 09:56:23 | watcher-api-scaling/5-cleanup-watcher
2026-03-14 09:56:26.625562 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:56:26.625567 | controller |       value -c Name -c Type\n    logger.go:42: 09:56:23 | watcher-api-scaling/5-cleanup-watcher
2026-03-14 09:56:26.625573 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:56:25 | watcher-api-scaling/5-cleanup-watcher
2026-03-14 09:56:26.625581 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:56:25 | watcher-api-scaling/5-cleanup-watcher
2026-03-14 09:56:26.625586 | controller |       | test step completed 5-cleanup-watcher\n    logger.go:42: 09:56:25 | watcher-api-scaling
2026-03-14 09:56:26.625592 | controller |       | skipping kubernetes event logging\n=== CONT  kuttl/harness/deps\n    logger.go:42:
2026-03-14 09:56:26.625615 | controller |       09:56:25 | deps | Ignoring infra.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n
2026-03-14 09:56:26.625621 | controller |       \   logger.go:42: 09:56:25 | deps | Ignoring keystone.yaml as it does not match
2026-03-14 09:56:26.625627 | controller |       file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n    logger.go:42: 09:56:25 |
2026-03-14 09:56:26.625633 | controller |       deps | Ignoring kustomization.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n
2026-03-14 09:56:26.625638 | controller |       \   logger.go:42: 09:56:25 | deps | Ignoring namespace.yaml as it does not match
2026-03-14 09:56:26.625644 | controller |       file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n    logger.go:42: 09:56:25 |
2026-03-14 09:56:26.625650 | controller |       deps | Ignoring telemetry.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n
2026-03-14 09:56:26.625656 | controller |       \   logger.go:42: 09:56:25 | deps | Skipping creation of user-supplied namespace:
2026-03-14 09:56:26.625662 | controller |       watcher-kuttl-default\n    logger.go:42: 09:56:25 | deps | skipping kubernetes event
2026-03-14 09:56:26.625667 | controller |       logging\n=== CONT  kuttl/harness/common\n    logger.go:42: 09:56:25 | common | Ignoring
2026-03-14 09:56:26.625676 | controller |       cleanup-assert.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n
2026-03-14 09:56:26.625690 | controller |       \   logger.go:42: 09:56:25 | common | Ignoring cleanup-errors.yaml as it does not
2026-03-14 09:56:26.625696 | controller |       match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n    logger.go:42: 09:56:25
2026-03-14 09:56:26.625702 | controller |       | common | Ignoring cleanup-watcher.yaml as it does not match file name regexp:
2026-03-14 09:56:26.625708 | controller |       ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n    logger.go:42: 09:56:25 | common | Ignoring
2026-03-14 09:56:26.625714 | controller |       deploy-with-defaults.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n
2026-03-14 09:56:26.625720 | controller |       \   logger.go:42: 09:56:25 | common | Skipping creation of user-supplied namespace:
2026-03-14 09:56:26.625725 | controller |       watcher-kuttl-default\n    logger.go:42: 09:56:25 | common | skipping kubernetes
2026-03-14 09:56:26.625731 | controller |       event logging\n=== NAME  kuttl\n    harness.go:406: run tests finished\n    harness.go:514:
2026-03-14 09:56:26.625741 | controller |       cleaning up\n    harness.go:571: removing temp folder: \"\"\n--- FAIL: kuttl (1500.34s)\n
2026-03-14 09:56:26.625747 | controller |       \   --- FAIL: kuttl/harness (0.00s)\n        --- FAIL: kuttl/harness/appcred-tests
2026-03-14 09:56:26.625753 | controller |       (956.65s)\n        --- PASS: kuttl/harness/watcher-notification (36.92s)\n        ---
2026-03-14 09:56:26.625758 | controller |       PASS: kuttl/harness/watcher-topology (38.12s)\n        --- PASS: kuttl/harness/watcher-tls-certs-change
2026-03-14 09:56:26.625764 | controller |       (42.17s)\n        --- PASS: kuttl/harness/watcher-tls (129.08s)\n        --- PASS:
2026-03-14 09:56:26.625770 | controller |       kuttl/harness/watcher-rmquser (35.43s)\n        --- PASS: kuttl/harness/watcher
2026-03-14 09:56:26.625776 | controller |       (74.95s)\n        --- PASS: kuttl/harness/watcher-cinder (128.08s)\n        ---
2026-03-14 09:56:26.625781 | controller |       PASS: kuttl/harness/watcher-api-scaling (58.90s)\n        --- PASS: kuttl/harness/deps
2026-03-14 09:56:26.625787 | controller |       (0.00s)\n        --- PASS: kuttl/harness/common (0.00s)\nFAIL"
2026-03-14 09:56:26.625792 | controller |     stdout_lines:
2026-03-14 09:56:26.625799 | controller |     - oc kuttl test --v 1 --start-kind=false --config test/kuttl/test-suites/default/config.yaml
2026-03-14 09:56:26.625804 | controller |     - === RUN   kuttl
2026-03-14 09:56:26.625810 | controller |     - '    harness.go:463: starting setup'
2026-03-14 09:56:26.625816 | controller |     - '    harness.go:255: running tests using configured kubeconfig.'
2026-03-14 09:56:26.625822 | controller |     - '    harness.go:278: Successful connection to cluster at: https://api.crc.testing:6443'
2026-03-14 09:56:26.625828 | controller |     - '    harness.go:363: running tests'
2026-03-14 09:56:26.625833 | controller |     - '    harness.go:75: going to run test suite with timeout of 300 seconds for each
2026-03-14 09:56:26.625839 | controller |       step'
2026-03-14 09:56:26.625845 | controller |     - '    harness.go:375: testsuite: test/kuttl/test-suites/default/ has 11 tests'
2026-03-14 09:56:26.625851 | controller |     - === RUN   kuttl/harness
2026-03-14 09:56:26.625857 | controller |     - === RUN   kuttl/harness/appcred-tests
2026-03-14 09:56:26.625862 | controller |     - === PAUSE kuttl/harness/appcred-tests
2026-03-14 09:56:26.625868 | controller |     - === RUN   kuttl/harness/common
2026-03-14 09:56:26.625874 | controller |     - === PAUSE kuttl/harness/common
2026-03-14 09:56:26.625879 | controller |     - === RUN   kuttl/harness/deps
2026-03-14 09:56:26.625885 | controller |     - === PAUSE kuttl/harness/deps
2026-03-14 09:56:26.625891 | controller |     - === RUN   kuttl/harness/watcher
2026-03-14 09:56:26.625896 | controller |     - === PAUSE kuttl/harness/watcher
2026-03-14 09:56:26.625902 | controller |     - === RUN   kuttl/harness/watcher-api-scaling
2026-03-14 09:56:26.625908 | controller |     - === PAUSE kuttl/harness/watcher-api-scaling
2026-03-14 09:56:26.625913 | controller |     - === RUN   kuttl/harness/watcher-cinder
2026-03-14 09:56:26.625919 | controller |     - === PAUSE kuttl/harness/watcher-cinder
2026-03-14 09:56:26.625924 | controller |     - === RUN   kuttl/harness/watcher-notification
2026-03-14 09:56:26.625930 | controller |     - === PAUSE kuttl/harness/watcher-notification
2026-03-14 09:56:26.625936 | controller |     - === RUN   kuttl/harness/watcher-rmquser
2026-03-14 09:56:26.625941 | controller |     - === PAUSE kuttl/harness/watcher-rmquser
2026-03-14 09:56:26.625947 | controller |     - === RUN   kuttl/harness/watcher-tls
2026-03-14 09:56:26.625953 | controller |     - === PAUSE kuttl/harness/watcher-tls
2026-03-14 09:56:26.625959 | controller |     - === RUN   kuttl/harness/watcher-tls-certs-change
2026-03-14 09:56:26.625965 | controller |     - === PAUSE kuttl/harness/watcher-tls-certs-change
2026-03-14 09:56:26.625971 | controller |     - === RUN   kuttl/harness/watcher-topology
2026-03-14 09:56:26.625976 | controller |     - === PAUSE kuttl/harness/watcher-topology
2026-03-14 09:56:26.625985 | controller |     - === CONT  kuttl/harness/appcred-tests
2026-03-14 09:56:26.625991 | controller |     - '    logger.go:42: 09:31:25 | appcred-tests | Skipping creation of user-supplied
2026-03-14 09:56:26.625997 | controller |       namespace: watcher-kuttl-default'
2026-03-14 09:56:26.626003 | controller |     - '    logger.go:42: 09:31:25 | appcred-tests/0-cleanup | starting test step 0-cleanup'
2026-03-14 09:56:26.626009 | controller |     - '    logger.go:42: 09:31:25 | appcred-tests/0-cleanup | test step completed 0-cleanup'
2026-03-14 09:56:26.626016 | controller |     - '    logger.go:42: 09:31:25 | appcred-tests/1-deploy | starting test step 1-deploy'
2026-03-14 09:56:26.626022 | controller |     - '    logger.go:42: 09:31:25 | appcred-tests/1-deploy | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:56:26.626028 | controller |       created'
2026-03-14 09:56:26.626036 | controller |     - '    logger.go:42: 09:31:25 | appcred-tests/1-deploy | running command: [sh -c set
2026-03-14 09:56:26.626042 | controller |       -euo pipefail'
2026-03-14 09:56:26.626048 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.626054 | controller |     - '        oc wait -n "${NS}" watcher watcher-kuttl --for=condition=Ready --timeout=600s]'
2026-03-14 09:56:26.626060 | controller |     - '    logger.go:42: 09:32:20 | appcred-tests/1-deploy | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:56:26.626066 | controller |       condition met'
2026-03-14 09:56:26.626080 | controller |     - '    logger.go:42: 09:32:20 | appcred-tests/1-deploy | test step completed 1-deploy'
2026-03-14 09:56:26.626089 | controller |     - '    logger.go:42: 09:32:20 | appcred-tests/2-deploy-appcred | starting test step
2026-03-14 09:56:26.626095 | controller |       2-deploy-appcred'
2026-03-14 09:56:26.626100 | controller |     - '    logger.go:42: 09:32:20 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.626106 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.626112 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.626117 | controller |     - '        '
2026-03-14 09:56:26.626123 | controller |     - '        old_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.626129 | controller |     - '        old_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.626135 | controller |     - '        old_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.626140 | controller |     - '        oc create configmap appcred-watcher-pre \'
2026-03-14 09:56:26.626146 | controller |     - '          --from-literal=api_uid="${old_api_uid}" \'
2026-03-14 09:56:26.626152 | controller |     - '          --from-literal=applier_uid="${old_applier_uid}" \'
2026-03-14 09:56:26.626157 | controller |     - '          --from-literal=decision_engine_uid="${old_de_uid}" \'
2026-03-14 09:56:26.626163 | controller |     - '          --dry-run=client -o yaml | oc apply -n "${NS}" -f -'
2026-03-14 09:56:26.626169 | controller |     - '        '
2026-03-14 09:56:26.626176 | controller |     - '        cat <<''EOF'' | oc apply -n "${NS}" -f -'
2026-03-14 09:56:26.626182 | controller |     - '        apiVersion: keystone.openstack.org/v1beta1'
2026-03-14 09:56:26.626187 | controller |     - '        kind: KeystoneApplicationCredential'
2026-03-14 09:56:26.626193 | controller |     - '        metadata:'
2026-03-14 09:56:26.626199 | controller |     - '          name: ac-watcher'
2026-03-14 09:56:26.626204 | controller |     - '        spec:'
2026-03-14 09:56:26.626210 | controller |     - '          secret: osp-secret'
2026-03-14 09:56:26.626216 | controller |     - '          passwordSelector: WatcherPassword'
2026-03-14 09:56:26.626221 | controller |     - '          userName: watcher'
2026-03-14 09:56:26.626227 | controller |     - '          roles:'
2026-03-14 09:56:26.626233 | controller |     - '            - admin'
2026-03-14 09:56:26.626239 | controller |     - '            - service'
2026-03-14 09:56:26.626244 | controller |     - '          unrestricted: false'
2026-03-14 09:56:26.626250 | controller |     - '        EOF'
2026-03-14 09:56:26.626256 | controller |     - '        '
2026-03-14 09:56:26.626261 | controller |     - '        oc patch watcher watcher-kuttl -n "${NS}" --type=merge -p ''{"spec":{"auth":{"applicationCredentialSecret":"ac-watcher-secret"}}}'']'
2026-03-14 09:56:26.626270 | controller |     - '    logger.go:42: 09:32:20 | appcred-tests/2-deploy-appcred | configmap/appcred-watcher-pre
2026-03-14 09:56:26.626276 | controller |       created'
2026-03-14 09:56:26.626282 | controller |     - '    logger.go:42: 09:32:21 | appcred-tests/2-deploy-appcred | keystoneapplicationcredential.keystone.openstack.org/ac-watcher
2026-03-14 09:56:26.626287 | controller |       created'
2026-03-14 09:56:26.626293 | controller |     - '    logger.go:42: 09:32:21 | appcred-tests/2-deploy-appcred | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:56:26.626302 | controller |       patched'
2026-03-14 09:56:26.626308 | controller |     - '    logger.go:42: 09:32:21 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.626314 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.626320 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.626325 | controller |     - '        '
2026-03-14 09:56:26.626331 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.626337 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.626343 | controller |     - '        '
2026-03-14 09:56:26.626349 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.626354 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.626360 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.626366 | controller |     - '          exit 1'
2026-03-14 09:56:26.626371 | controller |     - '        fi'
2026-03-14 09:56:26.626377 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.626383 | controller |     - '        '
2026-03-14 09:56:26.626389 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.626394 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.626400 | controller |       | base64 -d)'
2026-03-14 09:56:26.626405 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.626411 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.626417 | controller |     - '          exit 1'
2026-03-14 09:56:26.626422 | controller |     - '        fi'
2026-03-14 09:56:26.626428 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.626434 | controller |     - '        '
2026-03-14 09:56:26.626440 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.626445 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.626451 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.626457 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.626462 | controller |       --timeout=300s'
2026-03-14 09:56:26.626468 | controller |     - '        '
2026-03-14 09:56:26.626474 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.626479 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.626485 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.626491 | controller |     - '        '
2026-03-14 09:56:26.626497 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.626511 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.626517 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.626523 | controller |     - '        '
2026-03-14 09:56:26.626529 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.626535 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.626541 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.626547 | controller |       available"'
2026-03-14 09:56:26.626552 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.626558 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.626564 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.626572 | controller |     - '          exit 1'
2026-03-14 09:56:26.626578 | controller |     - '        fi'
2026-03-14 09:56:26.626584 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.626590 | controller |     - '        '
2026-03-14 09:56:26.626608 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.626616 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.626622 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.626628 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.626634 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.626640 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.626645 | controller |       -- \'
2026-03-14 09:56:26.626651 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.626657 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.626663 | controller |     - '    logger.go:42: 09:32:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.626669 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.626674 | controller |     - '    logger.go:42: 09:37:21 | appcred-tests/2-deploy-appcred | error: timed out
2026-03-14 09:56:26.626680 | controller |       waiting for the condition on keystoneapplicationcredentials/ac-watcher'
2026-03-14 09:56:26.626686 | controller |     - '    logger.go:42: 09:37:22 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.626693 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.626698 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.626704 | controller |     - '        '
2026-03-14 09:56:26.626710 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.626715 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.626721 | controller |     - '        '
2026-03-14 09:56:26.626727 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.626732 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.626738 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.626745 | controller |     - '          exit 1'
2026-03-14 09:56:26.626750 | controller |     - '        fi'
2026-03-14 09:56:26.626756 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.626762 | controller |     - '        '
2026-03-14 09:56:26.626768 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.626774 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.626779 | controller |       | base64 -d)'
2026-03-14 09:56:26.626785 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.626791 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.626797 | controller |     - '          exit 1'
2026-03-14 09:56:26.626802 | controller |     - '        fi'
2026-03-14 09:56:26.626808 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.626814 | controller |     - '        '
2026-03-14 09:56:26.626820 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.626826 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.626831 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.626837 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.626843 | controller |       --timeout=300s'
2026-03-14 09:56:26.626851 | controller |     - '        '
2026-03-14 09:56:26.626858 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.626863 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.626869 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.626875 | controller |     - '        '
2026-03-14 09:56:26.626881 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.626897 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.626903 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.626909 | controller |     - '        '
2026-03-14 09:56:26.626915 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.626921 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.626926 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.626932 | controller |       available"'
2026-03-14 09:56:26.626938 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.626954 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.626961 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.626966 | controller |     - '          exit 1'
2026-03-14 09:56:26.626972 | controller |     - '        fi'
2026-03-14 09:56:26.626978 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.626984 | controller |     - '        '
2026-03-14 09:56:26.626990 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.626996 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.627001 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.627007 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.627013 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.627022 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.627028 | controller |       -- \'
2026-03-14 09:56:26.627034 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.627039 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.627045 | controller |     - '    logger.go:42: 09:37:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.627051 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.627057 | controller |     - '    logger.go:42: 09:37:22 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.627063 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.627068 | controller |       found'
2026-03-14 09:56:26.627074 | controller |     - '    logger.go:42: 09:37:23 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.627083 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.627088 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.627094 | controller |     - '        '
2026-03-14 09:56:26.627100 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.627106 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.627112 | controller |     - '        '
2026-03-14 09:56:26.627119 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.627127 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.627133 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.627139 | controller |     - '          exit 1'
2026-03-14 09:56:26.627145 | controller |     - '        fi'
2026-03-14 09:56:26.627150 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.627156 | controller |     - '        '
2026-03-14 09:56:26.627162 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.627168 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.627174 | controller |       | base64 -d)'
2026-03-14 09:56:26.627180 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.627185 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.627191 | controller |     - '          exit 1'
2026-03-14 09:56:26.627198 | controller |     - '        fi'
2026-03-14 09:56:26.627204 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.627210 | controller |     - '        '
2026-03-14 09:56:26.627216 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.627221 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.627227 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.627233 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.627239 | controller |       --timeout=300s'
2026-03-14 09:56:26.627245 | controller |     - '        '
2026-03-14 09:56:26.627250 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.627256 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.627262 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.627268 | controller |     - '        '
2026-03-14 09:56:26.627274 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.627279 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.627297 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.627304 | controller |     - '        '
2026-03-14 09:56:26.627309 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.627315 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.627321 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.627327 | controller |       available"'
2026-03-14 09:56:26.627333 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.627338 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.627344 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.627350 | controller |     - '          exit 1'
2026-03-14 09:56:26.627355 | controller |     - '        fi'
2026-03-14 09:56:26.627361 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.627367 | controller |     - '        '
2026-03-14 09:56:26.627372 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.627378 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.627384 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.627389 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.627400 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.627406 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.627412 | controller |       -- \'
2026-03-14 09:56:26.627418 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.627424 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.627430 | controller |     - '    logger.go:42: 09:37:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.627435 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.627441 | controller |     - '    logger.go:42: 09:37:23 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.627447 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.627453 | controller |       found'
2026-03-14 09:56:26.627458 | controller |     - '    logger.go:42: 09:37:24 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.627464 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.627470 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.627476 | controller |     - '        '
2026-03-14 09:56:26.627481 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.627487 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.627493 | controller |     - '        '
2026-03-14 09:56:26.627502 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.627507 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.627513 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.627519 | controller |     - '          exit 1'
2026-03-14 09:56:26.627525 | controller |     - '        fi'
2026-03-14 09:56:26.627530 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.627536 | controller |     - '        '
2026-03-14 09:56:26.627542 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.627551 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.627559 | controller |       | base64 -d)'
2026-03-14 09:56:26.627565 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.627571 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.627577 | controller |     - '          exit 1'
2026-03-14 09:56:26.627582 | controller |     - '        fi'
2026-03-14 09:56:26.627588 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.627607 | controller |     - '        '
2026-03-14 09:56:26.627618 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.627624 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.627630 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.627635 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.627641 | controller |       --timeout=300s'
2026-03-14 09:56:26.627647 | controller |     - '        '
2026-03-14 09:56:26.627652 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.627658 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.627664 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.627670 | controller |     - '        '
2026-03-14 09:56:26.627675 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.627684 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.627698 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.627705 | controller |     - '        '
2026-03-14 09:56:26.627711 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.627716 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.627722 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.627728 | controller |       available"'
2026-03-14 09:56:26.627734 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.627739 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.627745 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.627751 | controller |     - '          exit 1'
2026-03-14 09:56:26.627757 | controller |     - '        fi'
2026-03-14 09:56:26.627763 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.627769 | controller |     - '        '
2026-03-14 09:56:26.627774 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.627780 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.627786 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.627791 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.627797 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.627803 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.627808 | controller |       -- \'
2026-03-14 09:56:26.627814 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.627820 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.627826 | controller |     - '    logger.go:42: 09:37:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.627832 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.627838 | controller |     - '    logger.go:42: 09:37:24 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.627844 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.627849 | controller |       found'
2026-03-14 09:56:26.627855 | controller |     - '    logger.go:42: 09:37:25 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.627861 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.627866 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.627872 | controller |     - '        '
2026-03-14 09:56:26.627878 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.627887 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.627892 | controller |     - '        '
2026-03-14 09:56:26.627898 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.627904 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.627910 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.627915 | controller |     - '          exit 1'
2026-03-14 09:56:26.627921 | controller |     - '        fi'
2026-03-14 09:56:26.627927 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.627933 | controller |     - '        '
2026-03-14 09:56:26.627938 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.627947 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.627953 | controller |       | base64 -d)'
2026-03-14 09:56:26.627959 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.627964 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.627970 | controller |     - '          exit 1'
2026-03-14 09:56:26.627975 | controller |     - '        fi'
2026-03-14 09:56:26.627981 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.627987 | controller |     - '        '
2026-03-14 09:56:26.627993 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.627999 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.628005 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.628010 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.628016 | controller |       --timeout=300s'
2026-03-14 09:56:26.628022 | controller |     - '        '
2026-03-14 09:56:26.628027 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.628033 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.628039 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.628045 | controller |     - '        '
2026-03-14 09:56:26.628051 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.628057 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.628062 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.628068 | controller |     - '        '
2026-03-14 09:56:26.628083 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.628089 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.628095 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.628101 | controller |       available"'
2026-03-14 09:56:26.628107 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.628112 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.628118 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.628124 | controller |     - '          exit 1'
2026-03-14 09:56:26.628129 | controller |     - '        fi'
2026-03-14 09:56:26.628135 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.628141 | controller |     - '        '
2026-03-14 09:56:26.628147 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.628152 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.628158 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.628164 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.628170 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.628176 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.628182 | controller |       -- \'
2026-03-14 09:56:26.628187 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.628193 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.628201 | controller |     - '    logger.go:42: 09:37:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.628207 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.628213 | controller |     - '    logger.go:42: 09:37:25 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.628219 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.628225 | controller |       found'
2026-03-14 09:56:26.628230 | controller |     - '    logger.go:42: 09:37:26 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.628236 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.628242 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.628248 | controller |     - '        '
2026-03-14 09:56:26.628254 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.628260 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.628266 | controller |     - '        '
2026-03-14 09:56:26.628271 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.628277 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.628283 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.628289 | controller |     - '          exit 1'
2026-03-14 09:56:26.628294 | controller |     - '        fi'
2026-03-14 09:56:26.628300 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.628306 | controller |     - '        '
2026-03-14 09:56:26.628312 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.628317 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.628323 | controller |       | base64 -d)'
2026-03-14 09:56:26.628329 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.628336 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.628343 | controller |     - '          exit 1'
2026-03-14 09:56:26.628350 | controller |     - '        fi'
2026-03-14 09:56:26.628357 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.628364 | controller |     - '        '
2026-03-14 09:56:26.628373 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.628379 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.628385 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.628391 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.628397 | controller |       --timeout=300s'
2026-03-14 09:56:26.628403 | controller |     - '        '
2026-03-14 09:56:26.628408 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.628414 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.628420 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.628426 | controller |     - '        '
2026-03-14 09:56:26.628431 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.628437 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.628445 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.628451 | controller |     - '        '
2026-03-14 09:56:26.628465 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.628474 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.628482 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.628488 | controller |       available"'
2026-03-14 09:56:26.628494 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.628500 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.628506 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.628512 | controller |     - '          exit 1'
2026-03-14 09:56:26.628518 | controller |     - '        fi'
2026-03-14 09:56:26.628524 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.628530 | controller |     - '        '
2026-03-14 09:56:26.628535 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.628541 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.628547 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.628553 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.628560 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.628566 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.628572 | controller |       -- \'
2026-03-14 09:56:26.628578 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.628584 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.628590 | controller |     - '    logger.go:42: 09:37:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.628611 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.628617 | controller |     - '    logger.go:42: 09:37:27 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.628623 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.628629 | controller |       found'
2026-03-14 09:56:26.628635 | controller |     - '    logger.go:42: 09:37:28 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.628641 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.628647 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.628653 | controller |     - '        '
2026-03-14 09:56:26.628659 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.628664 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.628673 | controller |     - '        '
2026-03-14 09:56:26.628679 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.628685 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.628691 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.628697 | controller |     - '          exit 1'
2026-03-14 09:56:26.628703 | controller |     - '        fi'
2026-03-14 09:56:26.628708 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.628714 | controller |     - '        '
2026-03-14 09:56:26.628720 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.628726 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.628731 | controller |       | base64 -d)'
2026-03-14 09:56:26.628737 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.628743 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.628749 | controller |     - '          exit 1'
2026-03-14 09:56:26.628754 | controller |     - '        fi'
2026-03-14 09:56:26.628760 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.628769 | controller |     - '        '
2026-03-14 09:56:26.628775 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.628781 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.628786 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.628792 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.628798 | controller |       --timeout=300s'
2026-03-14 09:56:26.628804 | controller |     - '        '
2026-03-14 09:56:26.628810 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.628816 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.628822 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.628828 | controller |     - '        '
2026-03-14 09:56:26.628834 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.628839 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.628845 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.628851 | controller |     - '        '
2026-03-14 09:56:26.628857 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.628872 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.628878 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.628884 | controller |       available"'
2026-03-14 09:56:26.628890 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.628896 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.628902 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.628908 | controller |     - '          exit 1'
2026-03-14 09:56:26.628914 | controller |     - '        fi'
2026-03-14 09:56:26.628919 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.628925 | controller |     - '        '
2026-03-14 09:56:26.628931 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.628936 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.628942 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.628948 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.628954 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.628960 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.628966 | controller |       -- \'
2026-03-14 09:56:26.628972 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.628977 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.628983 | controller |     - '    logger.go:42: 09:37:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.628989 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.628995 | controller |     - '    logger.go:42: 09:37:28 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.629000 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.629006 | controller |       found'
2026-03-14 09:56:26.629012 | controller |     - '    logger.go:42: 09:37:29 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.629020 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.629026 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.629032 | controller |     - '        '
2026-03-14 09:56:26.629038 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.629043 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.629049 | controller |     - '        '
2026-03-14 09:56:26.629055 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.629060 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.629066 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.629072 | controller |     - '          exit 1'
2026-03-14 09:56:26.629078 | controller |     - '        fi'
2026-03-14 09:56:26.629084 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.629089 | controller |     - '        '
2026-03-14 09:56:26.629095 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.629101 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.629107 | controller |       | base64 -d)'
2026-03-14 09:56:26.629112 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.629118 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.629124 | controller |     - '          exit 1'
2026-03-14 09:56:26.629130 | controller |     - '        fi'
2026-03-14 09:56:26.629135 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.629141 | controller |     - '        '
2026-03-14 09:56:26.629147 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.629153 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.629158 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.629164 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.629170 | controller |       --timeout=300s'
2026-03-14 09:56:26.629176 | controller |     - '        '
2026-03-14 09:56:26.629182 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.629187 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.629193 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.629199 | controller |     - '        '
2026-03-14 09:56:26.629205 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.629211 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.629216 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.629222 | controller |     - '        '
2026-03-14 09:56:26.629228 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.629234 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.629248 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.629254 | controller |       available"'
2026-03-14 09:56:26.629260 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.629266 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.629276 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.629281 | controller |     - '          exit 1'
2026-03-14 09:56:26.629287 | controller |     - '        fi'
2026-03-14 09:56:26.629293 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.629303 | controller |     - '        '
2026-03-14 09:56:26.629309 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.629314 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.629320 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.629326 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.629332 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.629338 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.629346 | controller |       -- \'
2026-03-14 09:56:26.629352 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.629358 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.629364 | controller |     - '    logger.go:42: 09:37:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.629370 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.629375 | controller |     - '    logger.go:42: 09:37:29 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.629381 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.629387 | controller |       found'
2026-03-14 09:56:26.629393 | controller |     - '    logger.go:42: 09:37:30 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.629399 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.629404 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.629410 | controller |     - '        '
2026-03-14 09:56:26.629416 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.629422 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.629427 | controller |     - '        '
2026-03-14 09:56:26.629433 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.629442 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.629448 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.629453 | controller |     - '          exit 1'
2026-03-14 09:56:26.629459 | controller |     - '        fi'
2026-03-14 09:56:26.629465 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.629470 | controller |     - '        '
2026-03-14 09:56:26.629476 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.629482 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.629487 | controller |       | base64 -d)'
2026-03-14 09:56:26.629493 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.629499 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.629505 | controller |     - '          exit 1'
2026-03-14 09:56:26.629511 | controller |     - '        fi'
2026-03-14 09:56:26.629516 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.629522 | controller |     - '        '
2026-03-14 09:56:26.629528 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.629534 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.629539 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.629545 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.629551 | controller |       --timeout=300s'
2026-03-14 09:56:26.629556 | controller |     - '        '
2026-03-14 09:56:26.629562 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.629570 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.629576 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.629582 | controller |     - '        '
2026-03-14 09:56:26.629588 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.629607 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.629615 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.629621 | controller |     - '        '
2026-03-14 09:56:26.629627 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.629633 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.629647 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.629653 | controller |       available"'
2026-03-14 09:56:26.629659 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.629665 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.629670 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.629676 | controller |     - '          exit 1'
2026-03-14 09:56:26.629682 | controller |     - '        fi'
2026-03-14 09:56:26.629688 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.629693 | controller |     - '        '
2026-03-14 09:56:26.629699 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.629705 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.629710 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.629716 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.629722 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.629728 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.629734 | controller |       -- \'
2026-03-14 09:56:26.629739 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.629748 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.629754 | controller |     - '    logger.go:42: 09:37:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.629760 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.629766 | controller |     - '    logger.go:42: 09:37:30 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.629772 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.629778 | controller |       found'
2026-03-14 09:56:26.629784 | controller |     - '    logger.go:42: 09:37:31 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.629789 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.629795 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.629801 | controller |     - '        '
2026-03-14 09:56:26.629807 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.629813 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.629819 | controller |     - '        '
2026-03-14 09:56:26.629825 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.629833 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.629839 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.629845 | controller |     - '          exit 1'
2026-03-14 09:56:26.629851 | controller |     - '        fi'
2026-03-14 09:56:26.629859 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.629865 | controller |     - '        '
2026-03-14 09:56:26.629871 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.629877 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.629883 | controller |       | base64 -d)'
2026-03-14 09:56:26.629889 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.629895 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.629900 | controller |     - '          exit 1'
2026-03-14 09:56:26.629906 | controller |     - '        fi'
2026-03-14 09:56:26.629912 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.629918 | controller |     - '        '
2026-03-14 09:56:26.629924 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.629930 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.629936 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.629941 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.629947 | controller |       --timeout=300s'
2026-03-14 09:56:26.629953 | controller |     - '        '
2026-03-14 09:56:26.629959 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.629965 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.629971 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.629977 | controller |     - '        '
2026-03-14 09:56:26.629983 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.629988 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.629994 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.630000 | controller |     - '        '
2026-03-14 09:56:26.630006 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.630012 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.630018 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.630024 | controller |       available"'
2026-03-14 09:56:26.630038 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.630045 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.630051 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.630057 | controller |     - '          exit 1'
2026-03-14 09:56:26.630063 | controller |     - '        fi'
2026-03-14 09:56:26.630069 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.630075 | controller |     - '        '
2026-03-14 09:56:26.630081 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.630087 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.630093 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.630098 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.630106 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.630112 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.630118 | controller |       -- \'
2026-03-14 09:56:26.630124 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.630129 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.630135 | controller |     - '    logger.go:42: 09:37:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.630141 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.630147 | controller |     - '    logger.go:42: 09:37:31 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.630153 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.630159 | controller |       found'
2026-03-14 09:56:26.630164 | controller |     - '    logger.go:42: 09:37:32 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.630170 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.630179 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.630186 | controller |     - '        '
2026-03-14 09:56:26.630192 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.630197 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.630203 | controller |     - '        '
2026-03-14 09:56:26.630209 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.630215 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.630221 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.630227 | controller |     - '          exit 1'
2026-03-14 09:56:26.630232 | controller |     - '        fi'
2026-03-14 09:56:26.630238 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.630244 | controller |     - '        '
2026-03-14 09:56:26.630250 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.630256 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.630262 | controller |       | base64 -d)'
2026-03-14 09:56:26.630268 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.630273 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.630280 | controller |     - '          exit 1'
2026-03-14 09:56:26.630285 | controller |     - '        fi'
2026-03-14 09:56:26.630291 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.630297 | controller |     - '        '
2026-03-14 09:56:26.630303 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.630309 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.630318 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.630323 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.630329 | controller |       --timeout=300s'
2026-03-14 09:56:26.630335 | controller |     - '        '
2026-03-14 09:56:26.630341 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.630346 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.630352 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.630358 | controller |     - '        '
2026-03-14 09:56:26.630364 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.630370 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.630378 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.630384 | controller |     - '        '
2026-03-14 09:56:26.630390 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.630396 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.630402 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.630407 | controller |       available"'
2026-03-14 09:56:26.630413 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.630428 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.630435 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.630440 | controller |     - '          exit 1'
2026-03-14 09:56:26.630446 | controller |     - '        fi'
2026-03-14 09:56:26.630452 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.630458 | controller |     - '        '
2026-03-14 09:56:26.630464 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.630470 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.630475 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.630481 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.630487 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.630493 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.630499 | controller |       -- \'
2026-03-14 09:56:26.630505 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.630510 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.630517 | controller |     - '    logger.go:42: 09:37:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.630522 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.630528 | controller |     - '    logger.go:42: 09:37:33 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.630534 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.630540 | controller |       found'
2026-03-14 09:56:26.630546 | controller |     - '    logger.go:42: 09:37:34 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.630552 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.630558 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.630563 | controller |     - '        '
2026-03-14 09:56:26.630569 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.630575 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.630581 | controller |     - '        '
2026-03-14 09:56:26.630587 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.630606 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.630615 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.630622 | controller |     - '          exit 1'
2026-03-14 09:56:26.630628 | controller |     - '        fi'
2026-03-14 09:56:26.630633 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.630639 | controller |     - '        '
2026-03-14 09:56:26.630645 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.630651 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.630659 | controller |       | base64 -d)'
2026-03-14 09:56:26.630666 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.630671 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.630677 | controller |     - '          exit 1'
2026-03-14 09:56:26.630685 | controller |     - '        fi'
2026-03-14 09:56:26.630691 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.630697 | controller |     - '        '
2026-03-14 09:56:26.630703 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.630709 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.630714 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.630720 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.630726 | controller |       --timeout=300s'
2026-03-14 09:56:26.630732 | controller |     - '        '
2026-03-14 09:56:26.630738 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.630744 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.630750 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.630756 | controller |     - '        '
2026-03-14 09:56:26.630762 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.630768 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.630774 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.630780 | controller |     - '        '
2026-03-14 09:56:26.630785 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.630791 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.630797 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.630803 | controller |       available"'
2026-03-14 09:56:26.630809 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.630824 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.630830 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.630836 | controller |     - '          exit 1'
2026-03-14 09:56:26.630841 | controller |     - '        fi'
2026-03-14 09:56:26.630847 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.630853 | controller |     - '        '
2026-03-14 09:56:26.630859 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.630864 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.630870 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.630876 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.630882 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.630888 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.630894 | controller |       -- \'
2026-03-14 09:56:26.630900 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.630906 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.630914 | controller |     - '    logger.go:42: 09:37:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.630920 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.630926 | controller |     - '    logger.go:42: 09:37:34 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.630932 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.630938 | controller |       found'
2026-03-14 09:56:26.630944 | controller |     - '    logger.go:42: 09:37:35 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.630964 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.630969 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.630975 | controller |     - '        '
2026-03-14 09:56:26.630981 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.630987 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.630993 | controller |     - '        '
2026-03-14 09:56:26.630999 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.631004 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.631010 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.631016 | controller |     - '          exit 1'
2026-03-14 09:56:26.631022 | controller |     - '        fi'
2026-03-14 09:56:26.631028 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.631034 | controller |     - '        '
2026-03-14 09:56:26.631040 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.631049 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.631055 | controller |       | base64 -d)'
2026-03-14 09:56:26.631061 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.631066 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.631072 | controller |     - '          exit 1'
2026-03-14 09:56:26.631078 | controller |     - '        fi'
2026-03-14 09:56:26.631084 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.631090 | controller |     - '        '
2026-03-14 09:56:26.631096 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.631102 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.631108 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.631116 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.631122 | controller |       --timeout=300s'
2026-03-14 09:56:26.631128 | controller |     - '        '
2026-03-14 09:56:26.631134 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.631144 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.631150 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.631156 | controller |     - '        '
2026-03-14 09:56:26.631162 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.631167 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.631174 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.631182 | controller |     - '        '
2026-03-14 09:56:26.631188 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.631194 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.631199 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.631208 | controller |       available"'
2026-03-14 09:56:26.631214 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.631220 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.631236 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.631242 | controller |     - '          exit 1'
2026-03-14 09:56:26.631248 | controller |     - '        fi'
2026-03-14 09:56:26.631264 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.631270 | controller |     - '        '
2026-03-14 09:56:26.631276 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.631282 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.631288 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.631294 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.631300 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.631306 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.631312 | controller |       -- \'
2026-03-14 09:56:26.631318 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.631323 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.631329 | controller |     - '    logger.go:42: 09:37:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.631335 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.631341 | controller |     - '    logger.go:42: 09:37:35 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.631347 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.631353 | controller |       found'
2026-03-14 09:56:26.631359 | controller |     - '    logger.go:42: 09:37:36 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.631365 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.631371 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.631377 | controller |     - '        '
2026-03-14 09:56:26.631383 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.631389 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.631395 | controller |     - '        '
2026-03-14 09:56:26.631401 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.631407 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.631413 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.631418 | controller |     - '          exit 1'
2026-03-14 09:56:26.631424 | controller |     - '        fi'
2026-03-14 09:56:26.631430 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.631436 | controller |     - '        '
2026-03-14 09:56:26.631442 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.631448 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.631453 | controller |       | base64 -d)'
2026-03-14 09:56:26.631459 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.631465 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.631471 | controller |     - '          exit 1'
2026-03-14 09:56:26.631477 | controller |     - '        fi'
2026-03-14 09:56:26.631482 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.631488 | controller |     - '        '
2026-03-14 09:56:26.631494 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.631503 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.631509 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.631515 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.631521 | controller |       --timeout=300s'
2026-03-14 09:56:26.631527 | controller |     - '        '
2026-03-14 09:56:26.631533 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.631539 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.631544 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.631550 | controller |     - '        '
2026-03-14 09:56:26.631556 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.631562 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.631568 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.631573 | controller |     - '        '
2026-03-14 09:56:26.631579 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.631585 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.631591 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.631614 | controller |       available"'
2026-03-14 09:56:26.631620 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.631626 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.631632 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.631647 | controller |     - '          exit 1'
2026-03-14 09:56:26.631653 | controller |     - '        fi'
2026-03-14 09:56:26.631659 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.631665 | controller |     - '        '
2026-03-14 09:56:26.631671 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.631677 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.631682 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.631688 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.631694 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.631700 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.631709 | controller |       -- \'
2026-03-14 09:56:26.631715 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.631721 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.631727 | controller |     - '    logger.go:42: 09:37:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.631733 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.631738 | controller |     - '    logger.go:42: 09:37:36 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.631744 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.631750 | controller |       found'
2026-03-14 09:56:26.631756 | controller |     - '    logger.go:42: 09:37:37 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.631765 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.631771 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.631777 | controller |     - '        '
2026-03-14 09:56:26.631782 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.631788 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.631794 | controller |     - '        '
2026-03-14 09:56:26.631800 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.631806 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.631812 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.631817 | controller |     - '          exit 1'
2026-03-14 09:56:26.631823 | controller |     - '        fi'
2026-03-14 09:56:26.631829 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.631835 | controller |     - '        '
2026-03-14 09:56:26.631841 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.631847 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.631852 | controller |       | base64 -d)'
2026-03-14 09:56:26.631858 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.631864 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.631870 | controller |     - '          exit 1'
2026-03-14 09:56:26.631875 | controller |     - '        fi'
2026-03-14 09:56:26.631881 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.631887 | controller |     - '        '
2026-03-14 09:56:26.631892 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.631898 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.631904 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.631910 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.631919 | controller |       --timeout=300s'
2026-03-14 09:56:26.631925 | controller |     - '        '
2026-03-14 09:56:26.631930 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.631936 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.631942 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.631948 | controller |     - '        '
2026-03-14 09:56:26.631954 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.631960 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.631966 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.631972 | controller |     - '        '
2026-03-14 09:56:26.631978 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.631984 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.631989 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.631995 | controller |       available"'
2026-03-14 09:56:26.632001 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.632007 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.632013 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.632018 | controller |     - '          exit 1'
2026-03-14 09:56:26.632024 | controller |     - '        fi'
2026-03-14 09:56:26.632038 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.632047 | controller |     - '        '
2026-03-14 09:56:26.632053 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.632059 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.632065 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.632071 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.632076 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.632082 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.632088 | controller |       -- \'
2026-03-14 09:56:26.632094 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.632099 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.632105 | controller |     - '    logger.go:42: 09:37:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.632111 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.632117 | controller |     - '    logger.go:42: 09:37:37 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.632123 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.632129 | controller |       found'
2026-03-14 09:56:26.632134 | controller |     - '    logger.go:42: 09:37:38 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.632140 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.632146 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.632153 | controller |     - '        '
2026-03-14 09:56:26.632159 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.632164 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.632171 | controller |     - '        '
2026-03-14 09:56:26.632177 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.632183 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.632189 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.632194 | controller |     - '          exit 1'
2026-03-14 09:56:26.632200 | controller |     - '        fi'
2026-03-14 09:56:26.632206 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.632212 | controller |     - '        '
2026-03-14 09:56:26.632218 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.632223 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.632230 | controller |       | base64 -d)'
2026-03-14 09:56:26.632236 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.632242 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.632248 | controller |     - '          exit 1'
2026-03-14 09:56:26.632253 | controller |     - '        fi'
2026-03-14 09:56:26.632259 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.632265 | controller |     - '        '
2026-03-14 09:56:26.632271 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.632277 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.632283 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.632289 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.632295 | controller |       --timeout=300s'
2026-03-14 09:56:26.632301 | controller |     - '        '
2026-03-14 09:56:26.632307 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.632315 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.632321 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.632327 | controller |     - '        '
2026-03-14 09:56:26.632333 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.632339 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.632344 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.632350 | controller |     - '        '
2026-03-14 09:56:26.632357 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.632362 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.632368 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.632374 | controller |       available"'
2026-03-14 09:56:26.632380 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.632386 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.632391 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.632397 | controller |     - '          exit 1'
2026-03-14 09:56:26.632403 | controller |     - '        fi'
2026-03-14 09:56:26.632417 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.632424 | controller |     - '        '
2026-03-14 09:56:26.632430 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.632436 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.632441 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.632447 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.632453 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.632459 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.632465 | controller |       -- \'
2026-03-14 09:56:26.632471 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.632477 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.632483 | controller |     - '    logger.go:42: 09:37:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.632489 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.632495 | controller |     - '    logger.go:42: 09:37:38 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.632501 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.632506 | controller |       found'
2026-03-14 09:56:26.632512 | controller |     - '    logger.go:42: 09:37:39 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.632518 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.632524 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.632530 | controller |     - '        '
2026-03-14 09:56:26.632536 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.632541 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.632547 | controller |     - '        '
2026-03-14 09:56:26.632553 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.632559 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.632568 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.632574 | controller |     - '          exit 1'
2026-03-14 09:56:26.632579 | controller |     - '        fi'
2026-03-14 09:56:26.632585 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.632591 | controller |     - '        '
2026-03-14 09:56:26.632617 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.632623 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.632629 | controller |       | base64 -d)'
2026-03-14 09:56:26.632635 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.632641 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.632647 | controller |     - '          exit 1'
2026-03-14 09:56:26.632653 | controller |     - '        fi'
2026-03-14 09:56:26.632658 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.632664 | controller |     - '        '
2026-03-14 09:56:26.632670 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.632676 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.632682 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.632688 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.632694 | controller |       --timeout=300s'
2026-03-14 09:56:26.632703 | controller |     - '        '
2026-03-14 09:56:26.632709 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.632715 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.632723 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.632729 | controller |     - '        '
2026-03-14 09:56:26.632735 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.632741 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.632747 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.632753 | controller |     - '        '
2026-03-14 09:56:26.632759 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.632767 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.632773 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.632779 | controller |       available"'
2026-03-14 09:56:26.632785 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.632791 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.632799 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.632805 | controller |     - '          exit 1'
2026-03-14 09:56:26.632811 | controller |     - '        fi'
2026-03-14 09:56:26.632817 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.632823 | controller |     - '        '
2026-03-14 09:56:26.632838 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.632844 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.632850 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.632857 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.632862 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.632873 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.632879 | controller |       -- \'
2026-03-14 09:56:26.632885 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.632891 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.632896 | controller |     - '    logger.go:42: 09:37:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.632902 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.632908 | controller |     - '    logger.go:42: 09:37:40 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.632914 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.632920 | controller |       found'
2026-03-14 09:56:26.632926 | controller |     - '    logger.go:42: 09:37:41 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.632932 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.632938 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.632944 | controller |     - '        '
2026-03-14 09:56:26.632950 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.632956 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.632961 | controller |     - '        '
2026-03-14 09:56:26.632967 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.632973 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.632979 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.632985 | controller |     - '          exit 1'
2026-03-14 09:56:26.632991 | controller |     - '        fi'
2026-03-14 09:56:26.632997 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.633005 | controller |     - '        '
2026-03-14 09:56:26.633012 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.633018 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.633027 | controller |       | base64 -d)'
2026-03-14 09:56:26.633033 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.633039 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.633045 | controller |     - '          exit 1'
2026-03-14 09:56:26.633051 | controller |     - '        fi'
2026-03-14 09:56:26.633057 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.633062 | controller |     - '        '
2026-03-14 09:56:26.633068 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.633074 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.633080 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.633085 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.633091 | controller |       --timeout=300s'
2026-03-14 09:56:26.633097 | controller |     - '        '
2026-03-14 09:56:26.633103 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.633109 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.633191 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.633198 | controller |     - '        '
2026-03-14 09:56:26.633204 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.633209 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.633219 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.633225 | controller |     - '        '
2026-03-14 09:56:26.633231 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.633236 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.633242 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.633248 | controller |       available"'
2026-03-14 09:56:26.633254 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.633259 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.633265 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.633271 | controller |     - '          exit 1'
2026-03-14 09:56:26.633277 | controller |     - '        fi'
2026-03-14 09:56:26.633282 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.633288 | controller |     - '        '
2026-03-14 09:56:26.633304 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.633310 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.633316 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.633334 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.633340 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.633346 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.633351 | controller |       -- \'
2026-03-14 09:56:26.633357 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.633363 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.633369 | controller |     - '    logger.go:42: 09:37:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.633375 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.633380 | controller |     - '    logger.go:42: 09:37:41 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.633386 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.633392 | controller |       found'
2026-03-14 09:56:26.633398 | controller |     - '    logger.go:42: 09:37:42 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.633403 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.633409 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.633415 | controller |     - '        '
2026-03-14 09:56:26.633421 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.633427 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.633433 | controller |     - '        '
2026-03-14 09:56:26.633439 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.633445 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.633451 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.633457 | controller |     - '          exit 1'
2026-03-14 09:56:26.633463 | controller |     - '        fi'
2026-03-14 09:56:26.633468 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.633474 | controller |     - '        '
2026-03-14 09:56:26.633480 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.633486 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.633495 | controller |       | base64 -d)'
2026-03-14 09:56:26.633501 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.633507 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.633513 | controller |     - '          exit 1'
2026-03-14 09:56:26.633519 | controller |     - '        fi'
2026-03-14 09:56:26.633525 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.633530 | controller |     - '        '
2026-03-14 09:56:26.633536 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.633542 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.633548 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.633553 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.633559 | controller |       --timeout=300s'
2026-03-14 09:56:26.633565 | controller |     - '        '
2026-03-14 09:56:26.633571 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.633577 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.633583 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.633589 | controller |     - '        '
2026-03-14 09:56:26.633609 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.633618 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.633636 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.633642 | controller |     - '        '
2026-03-14 09:56:26.633648 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.633654 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.633660 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.633669 | controller |       available"'
2026-03-14 09:56:26.633675 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.633681 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.633686 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.633692 | controller |     - '          exit 1'
2026-03-14 09:56:26.633698 | controller |     - '        fi'
2026-03-14 09:56:26.633704 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.633710 | controller |     - '        '
2026-03-14 09:56:26.633716 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.633732 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.633739 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.633745 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.633751 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.633757 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.633763 | controller |       -- \'
2026-03-14 09:56:26.633769 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.633775 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.633781 | controller |     - '    logger.go:42: 09:37:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.633790 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.633795 | controller |     - '    logger.go:42: 09:37:42 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.633801 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.633807 | controller |       found'
2026-03-14 09:56:26.633813 | controller |     - '    logger.go:42: 09:37:43 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.633819 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.633825 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.633831 | controller |     - '        '
2026-03-14 09:56:26.633837 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.633842 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.633848 | controller |     - '        '
2026-03-14 09:56:26.633854 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.633860 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.633866 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.633872 | controller |     - '          exit 1'
2026-03-14 09:56:26.633877 | controller |     - '        fi'
2026-03-14 09:56:26.633883 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.633889 | controller |     - '        '
2026-03-14 09:56:26.633895 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.633901 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.633907 | controller |       | base64 -d)'
2026-03-14 09:56:26.633912 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.633918 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.633924 | controller |     - '          exit 1'
2026-03-14 09:56:26.633930 | controller |     - '        fi'
2026-03-14 09:56:26.633936 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.633942 | controller |     - '        '
2026-03-14 09:56:26.633948 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.633954 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.633965 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.634005 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.634012 | controller |       --timeout=300s'
2026-03-14 09:56:26.634018 | controller |     - '        '
2026-03-14 09:56:26.634024 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.634047 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.634055 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.634062 | controller |     - '        '
2026-03-14 09:56:26.634068 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.634074 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.634080 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.634086 | controller |     - '        '
2026-03-14 09:56:26.634096 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.634102 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.634108 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.634114 | controller |       available"'
2026-03-14 09:56:26.634128 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.634134 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.634140 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.634150 | controller |     - '          exit 1'
2026-03-14 09:56:26.634157 | controller |     - '        fi'
2026-03-14 09:56:26.634163 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.634172 | controller |     - '        '
2026-03-14 09:56:26.634179 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.634185 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.634199 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.634209 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.634216 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.634226 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.634234 | controller |       -- \'
2026-03-14 09:56:26.634240 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.634246 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.634253 | controller |     - '    logger.go:42: 09:37:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.634259 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.634265 | controller |     - '    logger.go:42: 09:37:43 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.634272 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.634278 | controller |       found'
2026-03-14 09:56:26.634283 | controller |     - '    logger.go:42: 09:37:44 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.634289 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.634295 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.634301 | controller |     - '        '
2026-03-14 09:56:26.634307 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.634312 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.634318 | controller |     - '        '
2026-03-14 09:56:26.634324 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.634330 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.634336 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.634342 | controller |     - '          exit 1'
2026-03-14 09:56:26.634348 | controller |     - '        fi'
2026-03-14 09:56:26.634354 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.634360 | controller |     - '        '
2026-03-14 09:56:26.634365 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.634371 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.634377 | controller |       | base64 -d)'
2026-03-14 09:56:26.634386 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.634392 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.634398 | controller |     - '          exit 1'
2026-03-14 09:56:26.634404 | controller |     - '        fi'
2026-03-14 09:56:26.634410 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.634415 | controller |     - '        '
2026-03-14 09:56:26.634428 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.634434 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.634443 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.634449 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.634454 | controller |       --timeout=300s'
2026-03-14 09:56:26.634460 | controller |     - '        '
2026-03-14 09:56:26.634466 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.634472 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.634478 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.634484 | controller |     - '        '
2026-03-14 09:56:26.634489 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.634496 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.634501 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.634507 | controller |     - '        '
2026-03-14 09:56:26.634513 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.634519 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.634525 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.634530 | controller |       available"'
2026-03-14 09:56:26.634536 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.634542 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.634548 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.634553 | controller |     - '          exit 1'
2026-03-14 09:56:26.634559 | controller |     - '        fi'
2026-03-14 09:56:26.634565 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.634574 | controller |     - '        '
2026-03-14 09:56:26.634579 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.634585 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.634619 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.634629 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.634635 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.634641 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.634647 | controller |       -- \'
2026-03-14 09:56:26.634653 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.634659 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.634665 | controller |     - '    logger.go:42: 09:37:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.634671 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.634677 | controller |     - '    logger.go:42: 09:37:44 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.634683 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.634688 | controller |       found'
2026-03-14 09:56:26.634694 | controller |     - '    logger.go:42: 09:37:45 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.634700 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.634706 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.634714 | controller |     - '        '
2026-03-14 09:56:26.634720 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.634726 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.634732 | controller |     - '        '
2026-03-14 09:56:26.634738 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.634744 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.634750 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.634756 | controller |     - '          exit 1'
2026-03-14 09:56:26.634762 | controller |     - '        fi'
2026-03-14 09:56:26.634768 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.634774 | controller |     - '        '
2026-03-14 09:56:26.634780 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.634789 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.634795 | controller |       | base64 -d)'
2026-03-14 09:56:26.634801 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.634807 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.634813 | controller |     - '          exit 1'
2026-03-14 09:56:26.634819 | controller |     - '        fi'
2026-03-14 09:56:26.634824 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.634831 | controller |     - '        '
2026-03-14 09:56:26.634837 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.634842 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.634848 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.634854 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.634860 | controller |       --timeout=300s'
2026-03-14 09:56:26.634865 | controller |     - '        '
2026-03-14 09:56:26.634871 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.634877 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.634883 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.634889 | controller |     - '        '
2026-03-14 09:56:26.634894 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.634901 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.634907 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.634912 | controller |     - '        '
2026-03-14 09:56:26.634918 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.634924 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.634930 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.634936 | controller |       available"'
2026-03-14 09:56:26.634942 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.634961 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.634967 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.634974 | controller |     - '          exit 1'
2026-03-14 09:56:26.634979 | controller |     - '        fi'
2026-03-14 09:56:26.634986 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.634992 | controller |     - '        '
2026-03-14 09:56:26.635000 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.635005 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.635020 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.635034 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.635040 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.635046 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.635057 | controller |       -- \'
2026-03-14 09:56:26.635063 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.635074 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.635080 | controller |     - '    logger.go:42: 09:37:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.635086 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.635092 | controller |     - '    logger.go:42: 09:37:46 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.635098 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.635104 | controller |       found'
2026-03-14 09:56:26.635109 | controller |     - '    logger.go:42: 09:37:47 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.635115 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.635121 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.635127 | controller |     - '        '
2026-03-14 09:56:26.635132 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.635138 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.635144 | controller |     - '        '
2026-03-14 09:56:26.635150 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.635159 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.635167 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.635174 | controller |     - '          exit 1'
2026-03-14 09:56:26.635180 | controller |     - '        fi'
2026-03-14 09:56:26.635186 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.635192 | controller |     - '        '
2026-03-14 09:56:26.635198 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.635204 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.635209 | controller |       | base64 -d)'
2026-03-14 09:56:26.635216 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.635221 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.635227 | controller |     - '          exit 1'
2026-03-14 09:56:26.635233 | controller |     - '        fi'
2026-03-14 09:56:26.635239 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.635245 | controller |     - '        '
2026-03-14 09:56:26.635251 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.635257 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.635262 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.635268 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.635274 | controller |       --timeout=300s'
2026-03-14 09:56:26.635279 | controller |     - '        '
2026-03-14 09:56:26.635285 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.635291 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.635299 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.635305 | controller |     - '        '
2026-03-14 09:56:26.635311 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.635317 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.635323 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.635328 | controller |     - '        '
2026-03-14 09:56:26.635334 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.635340 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.635345 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.635351 | controller |       available"'
2026-03-14 09:56:26.635357 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.635363 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.635369 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.635375 | controller |     - '          exit 1'
2026-03-14 09:56:26.635381 | controller |     - '        fi'
2026-03-14 09:56:26.635386 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.635392 | controller |     - '        '
2026-03-14 09:56:26.635398 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.635406 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.635412 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.635428 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.635435 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.635441 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.635446 | controller |       -- \'
2026-03-14 09:56:26.635452 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.635458 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.635464 | controller |     - '    logger.go:42: 09:37:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.635470 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.635475 | controller |     - '    logger.go:42: 09:37:47 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.635482 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.635487 | controller |       found'
2026-03-14 09:56:26.635493 | controller |     - '    logger.go:42: 09:37:48 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.635499 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.635505 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.635510 | controller |     - '        '
2026-03-14 09:56:26.635517 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.635522 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.635528 | controller |     - '        '
2026-03-14 09:56:26.635534 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.635540 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.635545 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.635554 | controller |     - '          exit 1'
2026-03-14 09:56:26.635560 | controller |     - '        fi'
2026-03-14 09:56:26.635566 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.635572 | controller |     - '        '
2026-03-14 09:56:26.635578 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.635584 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.635589 | controller |       | base64 -d)'
2026-03-14 09:56:26.635609 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.635627 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.635634 | controller |     - '          exit 1'
2026-03-14 09:56:26.635639 | controller |     - '        fi'
2026-03-14 09:56:26.635645 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.635651 | controller |     - '        '
2026-03-14 09:56:26.635657 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.635662 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.635668 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.635674 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.635680 | controller |       --timeout=300s'
2026-03-14 09:56:26.635685 | controller |     - '        '
2026-03-14 09:56:26.635691 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.635697 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.635702 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.635711 | controller |     - '        '
2026-03-14 09:56:26.635717 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.635723 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.635728 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.635735 | controller |     - '        '
2026-03-14 09:56:26.635740 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.635746 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.635752 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.635758 | controller |       available"'
2026-03-14 09:56:26.635764 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.635770 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.635775 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.635781 | controller |     - '          exit 1'
2026-03-14 09:56:26.635787 | controller |     - '        fi'
2026-03-14 09:56:26.635792 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.635798 | controller |     - '        '
2026-03-14 09:56:26.635804 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.635809 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.635815 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.635823 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.635838 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.635848 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.635853 | controller |       -- \'
2026-03-14 09:56:26.635859 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.635865 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.635871 | controller |     - '    logger.go:42: 09:37:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.635884 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.635890 | controller |     - '    logger.go:42: 09:37:48 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.635896 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.635902 | controller |       found'
2026-03-14 09:56:26.635907 | controller |     - '    logger.go:42: 09:37:49 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.635913 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.635919 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.635925 | controller |     - '        '
2026-03-14 09:56:26.635930 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.635936 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.635942 | controller |     - '        '
2026-03-14 09:56:26.635948 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.635954 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.635960 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.635965 | controller |     - '          exit 1'
2026-03-14 09:56:26.635971 | controller |     - '        fi'
2026-03-14 09:56:26.635977 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.635982 | controller |     - '        '
2026-03-14 09:56:26.635988 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.635994 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.636000 | controller |       | base64 -d)'
2026-03-14 09:56:26.636006 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.636012 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.636018 | controller |     - '          exit 1'
2026-03-14 09:56:26.636024 | controller |     - '        fi'
2026-03-14 09:56:26.636030 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.636035 | controller |     - '        '
2026-03-14 09:56:26.636041 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.636047 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.636053 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.636059 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.636065 | controller |       --timeout=300s'
2026-03-14 09:56:26.636071 | controller |     - '        '
2026-03-14 09:56:26.636076 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.636082 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.636088 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.636094 | controller |     - '        '
2026-03-14 09:56:26.636100 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.636106 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.636111 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.636120 | controller |     - '        '
2026-03-14 09:56:26.636126 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.636132 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.636138 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.636144 | controller |       available"'
2026-03-14 09:56:26.636149 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.636155 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.636161 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.636167 | controller |     - '          exit 1'
2026-03-14 09:56:26.636172 | controller |     - '        fi'
2026-03-14 09:56:26.636178 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.636184 | controller |     - '        '
2026-03-14 09:56:26.636190 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.636196 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.636202 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.636208 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.636226 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.636233 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.636239 | controller |       -- \'
2026-03-14 09:56:26.636244 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.636250 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.636256 | controller |     - '    logger.go:42: 09:37:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.636262 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.636268 | controller |     - '    logger.go:42: 09:37:49 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.636274 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.636280 | controller |       found'
2026-03-14 09:56:26.636286 | controller |     - '    logger.go:42: 09:37:50 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.636291 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.636297 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.636303 | controller |     - '        '
2026-03-14 09:56:26.636309 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.636315 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.636321 | controller |     - '        '
2026-03-14 09:56:26.636326 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.636332 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.636338 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.636344 | controller |     - '          exit 1'
2026-03-14 09:56:26.636350 | controller |     - '        fi'
2026-03-14 09:56:26.636355 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.636361 | controller |     - '        '
2026-03-14 09:56:26.636367 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.636373 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.636379 | controller |       | base64 -d)'
2026-03-14 09:56:26.636385 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.636393 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.636399 | controller |     - '          exit 1'
2026-03-14 09:56:26.636405 | controller |     - '        fi'
2026-03-14 09:56:26.636410 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.636416 | controller |     - '        '
2026-03-14 09:56:26.636422 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.636428 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.636434 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.636439 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.636445 | controller |       --timeout=300s'
2026-03-14 09:56:26.636451 | controller |     - '        '
2026-03-14 09:56:26.636457 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.636463 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.636469 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.636475 | controller |     - '        '
2026-03-14 09:56:26.636481 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.636487 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.636493 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.636499 | controller |     - '        '
2026-03-14 09:56:26.636505 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.636511 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.636517 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.636523 | controller |       available"'
2026-03-14 09:56:26.636529 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.636541 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.636550 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.636556 | controller |     - '          exit 1'
2026-03-14 09:56:26.636563 | controller |     - '        fi'
2026-03-14 09:56:26.636569 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.636574 | controller |     - '        '
2026-03-14 09:56:26.636580 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.636586 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.636605 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.636615 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.636628 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.636635 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.636641 | controller |       -- \'
2026-03-14 09:56:26.636646 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.636652 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.636658 | controller |     - '    logger.go:42: 09:37:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.636666 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.636672 | controller |     - '    logger.go:42: 09:37:50 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.636678 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.636684 | controller |       found'
2026-03-14 09:56:26.636690 | controller |     - '    logger.go:42: 09:37:51 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.636695 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.636701 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.636707 | controller |     - '        '
2026-03-14 09:56:26.636713 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.636719 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.636724 | controller |     - '        '
2026-03-14 09:56:26.636730 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.636736 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.636742 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.636748 | controller |     - '          exit 1'
2026-03-14 09:56:26.636754 | controller |     - '        fi'
2026-03-14 09:56:26.636760 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.636766 | controller |     - '        '
2026-03-14 09:56:26.636772 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.636777 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.636783 | controller |       | base64 -d)'
2026-03-14 09:56:26.636789 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.636795 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.636801 | controller |     - '          exit 1'
2026-03-14 09:56:26.636807 | controller |     - '        fi'
2026-03-14 09:56:26.636812 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.636818 | controller |     - '        '
2026-03-14 09:56:26.636824 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.636830 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.636836 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.636842 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.636848 | controller |       --timeout=300s'
2026-03-14 09:56:26.636854 | controller |     - '        '
2026-03-14 09:56:26.636859 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.636869 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.636874 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.636880 | controller |     - '        '
2026-03-14 09:56:26.636886 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.636891 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.636897 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.636903 | controller |     - '        '
2026-03-14 09:56:26.636909 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.636915 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.636921 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.636927 | controller |       available"'
2026-03-14 09:56:26.636933 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.636943 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.636948 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.636954 | controller |     - '          exit 1'
2026-03-14 09:56:26.636960 | controller |     - '        fi'
2026-03-14 09:56:26.636966 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.636972 | controller |     - '        '
2026-03-14 09:56:26.636978 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.636984 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.636990 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.636996 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.637002 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.637021 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.637028 | controller |       -- \'
2026-03-14 09:56:26.637034 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.637040 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.637046 | controller |     - '    logger.go:42: 09:37:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.637052 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.637058 | controller |     - '    logger.go:42: 09:37:51 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.637063 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.637069 | controller |       found'
2026-03-14 09:56:26.637074 | controller |     - '    logger.go:42: 09:37:52 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.637080 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.637086 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.637092 | controller |     - '        '
2026-03-14 09:56:26.637098 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.637103 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.637109 | controller |     - '        '
2026-03-14 09:56:26.637115 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.637121 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.637126 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.637132 | controller |     - '          exit 1'
2026-03-14 09:56:26.637138 | controller |     - '        fi'
2026-03-14 09:56:26.637144 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.637149 | controller |     - '        '
2026-03-14 09:56:26.637155 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.637161 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.637167 | controller |       | base64 -d)'
2026-03-14 09:56:26.637173 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.637179 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.637184 | controller |     - '          exit 1'
2026-03-14 09:56:26.637190 | controller |     - '        fi'
2026-03-14 09:56:26.637196 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.637201 | controller |     - '        '
2026-03-14 09:56:26.637207 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.637213 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.637221 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.637227 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.637233 | controller |       --timeout=300s'
2026-03-14 09:56:26.637239 | controller |     - '        '
2026-03-14 09:56:26.637245 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.637250 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.637256 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.637262 | controller |     - '        '
2026-03-14 09:56:26.637268 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.637274 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.637280 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.637288 | controller |     - '        '
2026-03-14 09:56:26.637294 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.637300 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.637306 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.637311 | controller |       available"'
2026-03-14 09:56:26.637317 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.637324 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.637329 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.637335 | controller |     - '          exit 1'
2026-03-14 09:56:26.637341 | controller |     - '        fi'
2026-03-14 09:56:26.637347 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.637352 | controller |     - '        '
2026-03-14 09:56:26.637358 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.637364 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.637370 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.637376 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.637381 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.637387 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.637403 | controller |       -- \'
2026-03-14 09:56:26.637410 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.637416 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.637422 | controller |     - '    logger.go:42: 09:37:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.637428 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.637434 | controller |     - '    logger.go:42: 09:37:53 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.637440 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.637446 | controller |       found'
2026-03-14 09:56:26.637452 | controller |     - '    logger.go:42: 09:37:54 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.637457 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.637463 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.637470 | controller |     - '        '
2026-03-14 09:56:26.637479 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.637485 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.637491 | controller |     - '        '
2026-03-14 09:56:26.637497 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.637503 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.637508 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.637514 | controller |     - '          exit 1'
2026-03-14 09:56:26.637520 | controller |     - '        fi'
2026-03-14 09:56:26.637526 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.637532 | controller |     - '        '
2026-03-14 09:56:26.637537 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.637543 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.637549 | controller |       | base64 -d)'
2026-03-14 09:56:26.637554 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.637560 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.637567 | controller |     - '          exit 1'
2026-03-14 09:56:26.637572 | controller |     - '        fi'
2026-03-14 09:56:26.637578 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.637584 | controller |     - '        '
2026-03-14 09:56:26.637590 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.637618 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.637624 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.637630 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.637637 | controller |       --timeout=300s'
2026-03-14 09:56:26.637643 | controller |     - '        '
2026-03-14 09:56:26.637649 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.637655 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.637661 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.637667 | controller |     - '        '
2026-03-14 09:56:26.637672 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.637678 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.637684 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.637690 | controller |     - '        '
2026-03-14 09:56:26.637695 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.637701 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.637707 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.637713 | controller |       available"'
2026-03-14 09:56:26.637719 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.637725 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.637731 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.637737 | controller |     - '          exit 1'
2026-03-14 09:56:26.637743 | controller |     - '        fi'
2026-03-14 09:56:26.637749 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.637755 | controller |     - '        '
2026-03-14 09:56:26.637761 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.637770 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.637776 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.637782 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.637788 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.637793 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.637799 | controller |       -- \'
2026-03-14 09:56:26.637815 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.637821 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.637827 | controller |     - '    logger.go:42: 09:37:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.637833 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.637839 | controller |     - '    logger.go:42: 09:37:54 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.637849 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.637855 | controller |       found'
2026-03-14 09:56:26.637860 | controller |     - '    logger.go:42: 09:37:55 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.637867 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.637873 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.637878 | controller |     - '        '
2026-03-14 09:56:26.637884 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.637890 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.637896 | controller |     - '        '
2026-03-14 09:56:26.637902 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.637908 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.637914 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.637920 | controller |     - '          exit 1'
2026-03-14 09:56:26.637926 | controller |     - '        fi'
2026-03-14 09:56:26.637932 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.637938 | controller |     - '        '
2026-03-14 09:56:26.637944 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.637949 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.637955 | controller |       | base64 -d)'
2026-03-14 09:56:26.637961 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.637967 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.637973 | controller |     - '          exit 1'
2026-03-14 09:56:26.637979 | controller |     - '        fi'
2026-03-14 09:56:26.637985 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.637991 | controller |     - '        '
2026-03-14 09:56:26.637997 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.638002 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.638008 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.638014 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.638020 | controller |       --timeout=300s'
2026-03-14 09:56:26.638026 | controller |     - '        '
2026-03-14 09:56:26.638032 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.638038 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.638046 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.638052 | controller |     - '        '
2026-03-14 09:56:26.638058 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.638063 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.638069 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.638075 | controller |     - '        '
2026-03-14 09:56:26.638081 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.638087 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.638093 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.638098 | controller |       available"'
2026-03-14 09:56:26.638104 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.638110 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.638116 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.638122 | controller |     - '          exit 1'
2026-03-14 09:56:26.638127 | controller |     - '        fi'
2026-03-14 09:56:26.638133 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.638139 | controller |     - '        '
2026-03-14 09:56:26.638145 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.638151 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.638156 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.638162 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.638168 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.638174 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.638180 | controller |       -- \'
2026-03-14 09:56:26.638194 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.638200 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.638206 | controller |     - '    logger.go:42: 09:37:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.638212 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.638219 | controller |     - '    logger.go:42: 09:37:55 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.638225 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.638230 | controller |       found'
2026-03-14 09:56:26.638236 | controller |     - '    logger.go:42: 09:37:56 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.638242 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.638248 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.638254 | controller |     - '        '
2026-03-14 09:56:26.638260 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.638266 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.638272 | controller |     - '        '
2026-03-14 09:56:26.638277 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.638283 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.638289 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.638297 | controller |     - '          exit 1'
2026-03-14 09:56:26.638303 | controller |     - '        fi'
2026-03-14 09:56:26.638309 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.638315 | controller |     - '        '
2026-03-14 09:56:26.638321 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.638330 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.638336 | controller |       | base64 -d)'
2026-03-14 09:56:26.638342 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.638348 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.638354 | controller |     - '          exit 1'
2026-03-14 09:56:26.638360 | controller |     - '        fi'
2026-03-14 09:56:26.638366 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.638372 | controller |     - '        '
2026-03-14 09:56:26.638377 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.638385 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.638391 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.638397 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.638403 | controller |       --timeout=300s'
2026-03-14 09:56:26.638409 | controller |     - '        '
2026-03-14 09:56:26.638415 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.638421 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.638427 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.638433 | controller |     - '        '
2026-03-14 09:56:26.638438 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.638444 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.638453 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.638459 | controller |     - '        '
2026-03-14 09:56:26.638465 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.638471 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.638477 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.638483 | controller |       available"'
2026-03-14 09:56:26.638489 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.638495 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.638500 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.638506 | controller |     - '          exit 1'
2026-03-14 09:56:26.638513 | controller |     - '        fi'
2026-03-14 09:56:26.638519 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.638525 | controller |     - '        '
2026-03-14 09:56:26.638531 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.638536 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.638542 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.638548 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.638554 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.638560 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.638568 | controller |       -- \'
2026-03-14 09:56:26.638574 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.638589 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.638612 | controller |     - '    logger.go:42: 09:37:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.638618 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.638625 | controller |     - '    logger.go:42: 09:37:56 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.638630 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.638640 | controller |       found'
2026-03-14 09:56:26.638646 | controller |     - '    logger.go:42: 09:37:57 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.638652 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.638657 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.638663 | controller |     - '        '
2026-03-14 09:56:26.638669 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.638675 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.638681 | controller |     - '        '
2026-03-14 09:56:26.638687 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.638693 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.638698 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.638704 | controller |     - '          exit 1'
2026-03-14 09:56:26.638710 | controller |     - '        fi'
2026-03-14 09:56:26.638716 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.638727 | controller |     - '        '
2026-03-14 09:56:26.638733 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.638739 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.638747 | controller |       | base64 -d)'
2026-03-14 09:56:26.638753 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.638759 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.638765 | controller |     - '          exit 1'
2026-03-14 09:56:26.638771 | controller |     - '        fi'
2026-03-14 09:56:26.638779 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.638785 | controller |     - '        '
2026-03-14 09:56:26.638791 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.638797 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.638803 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.638809 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.638815 | controller |       --timeout=300s'
2026-03-14 09:56:26.638821 | controller |     - '        '
2026-03-14 09:56:26.638827 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.638833 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.638839 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.638845 | controller |     - '        '
2026-03-14 09:56:26.638850 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.638856 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.638862 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.638870 | controller |     - '        '
2026-03-14 09:56:26.638876 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.638882 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.638888 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.638894 | controller |       available"'
2026-03-14 09:56:26.638900 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.638906 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.638912 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.638917 | controller |     - '          exit 1'
2026-03-14 09:56:26.638923 | controller |     - '        fi'
2026-03-14 09:56:26.638929 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.638935 | controller |     - '        '
2026-03-14 09:56:26.638941 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.638947 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.638967 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.638973 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.638979 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.638985 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.638991 | controller |       -- \'
2026-03-14 09:56:26.638997 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.639012 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.639019 | controller |     - '    logger.go:42: 09:37:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.639025 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.639031 | controller |     - '    logger.go:42: 09:37:57 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.639036 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.639042 | controller |       found'
2026-03-14 09:56:26.639048 | controller |     - '    logger.go:42: 09:37:58 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.639054 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.639060 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.639066 | controller |     - '        '
2026-03-14 09:56:26.639072 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.639078 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.639084 | controller |     - '        '
2026-03-14 09:56:26.639090 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.639095 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.639101 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.639107 | controller |     - '          exit 1'
2026-03-14 09:56:26.639113 | controller |     - '        fi'
2026-03-14 09:56:26.639119 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.639124 | controller |     - '        '
2026-03-14 09:56:26.639130 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.639136 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.639142 | controller |       | base64 -d)'
2026-03-14 09:56:26.639148 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.639154 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.639163 | controller |     - '          exit 1'
2026-03-14 09:56:26.639168 | controller |     - '        fi'
2026-03-14 09:56:26.639174 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.639180 | controller |     - '        '
2026-03-14 09:56:26.639186 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.639191 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.639197 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.639203 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.639209 | controller |       --timeout=300s'
2026-03-14 09:56:26.639214 | controller |     - '        '
2026-03-14 09:56:26.639220 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.639226 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.639232 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.639238 | controller |     - '        '
2026-03-14 09:56:26.639243 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.639249 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.639255 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.639261 | controller |     - '        '
2026-03-14 09:56:26.639267 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.639272 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.639278 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.639284 | controller |       available"'
2026-03-14 09:56:26.639290 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.639295 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.639301 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.639307 | controller |     - '          exit 1'
2026-03-14 09:56:26.639313 | controller |     - '        fi'
2026-03-14 09:56:26.639319 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.639325 | controller |     - '        '
2026-03-14 09:56:26.639331 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.639336 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.639342 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.639348 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.639354 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.639360 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.639366 | controller |       -- \'
2026-03-14 09:56:26.639371 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.639378 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.639393 | controller |     - '    logger.go:42: 09:37:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.639399 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.639405 | controller |     - '    logger.go:42: 09:37:58 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.639414 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.639419 | controller |       found'
2026-03-14 09:56:26.639425 | controller |     - '    logger.go:42: 09:37:59 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.639434 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.639440 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.639446 | controller |     - '        '
2026-03-14 09:56:26.639452 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.639458 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.639464 | controller |     - '        '
2026-03-14 09:56:26.639470 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.639476 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.639482 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.639488 | controller |     - '          exit 1'
2026-03-14 09:56:26.639494 | controller |     - '        fi'
2026-03-14 09:56:26.639500 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.639506 | controller |     - '        '
2026-03-14 09:56:26.639512 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.639517 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.639523 | controller |       | base64 -d)'
2026-03-14 09:56:26.639529 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.639535 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.639541 | controller |     - '          exit 1'
2026-03-14 09:56:26.639546 | controller |     - '        fi'
2026-03-14 09:56:26.639552 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.639558 | controller |     - '        '
2026-03-14 09:56:26.639564 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.639570 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.639576 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.639582 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.639588 | controller |       --timeout=300s'
2026-03-14 09:56:26.639606 | controller |     - '        '
2026-03-14 09:56:26.639615 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.639621 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.639627 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.639633 | controller |     - '        '
2026-03-14 09:56:26.639639 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.639645 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.639651 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.639657 | controller |     - '        '
2026-03-14 09:56:26.639663 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.639669 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.639675 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.639681 | controller |       available"'
2026-03-14 09:56:26.639687 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.639693 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.639701 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.639707 | controller |     - '          exit 1'
2026-03-14 09:56:26.639713 | controller |     - '        fi'
2026-03-14 09:56:26.639719 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.639725 | controller |     - '        '
2026-03-14 09:56:26.639731 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.639737 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.639743 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.639748 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.639754 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.639760 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.639766 | controller |       -- \'
2026-03-14 09:56:26.639772 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.639781 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.639796 | controller |     - '    logger.go:42: 09:37:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.639802 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.639808 | controller |     - '    logger.go:42: 09:37:59 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.639814 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.639820 | controller |       found'
2026-03-14 09:56:26.639826 | controller |     - '    logger.go:42: 09:38:00 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.639832 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.639838 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.639844 | controller |     - '        '
2026-03-14 09:56:26.639850 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.639856 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.639862 | controller |     - '        '
2026-03-14 09:56:26.639868 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.639873 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.639879 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.639886 | controller |     - '          exit 1'
2026-03-14 09:56:26.639891 | controller |     - '        fi'
2026-03-14 09:56:26.639897 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.639903 | controller |     - '        '
2026-03-14 09:56:26.639909 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.639915 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.639921 | controller |       | base64 -d)'
2026-03-14 09:56:26.639927 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.639932 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.639938 | controller |     - '          exit 1'
2026-03-14 09:56:26.639944 | controller |     - '        fi'
2026-03-14 09:56:26.639950 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.639955 | controller |     - '        '
2026-03-14 09:56:26.639961 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.639967 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.639973 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.639981 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.639987 | controller |       --timeout=300s'
2026-03-14 09:56:26.639993 | controller |     - '        '
2026-03-14 09:56:26.639999 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.640005 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.640011 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.640017 | controller |     - '        '
2026-03-14 09:56:26.640023 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.640028 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.640034 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.640040 | controller |     - '        '
2026-03-14 09:56:26.640046 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.640052 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.640057 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.640063 | controller |       available"'
2026-03-14 09:56:26.640069 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.640075 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.640084 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.640089 | controller |     - '          exit 1'
2026-03-14 09:56:26.640095 | controller |     - '        fi'
2026-03-14 09:56:26.640101 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.640107 | controller |     - '        '
2026-03-14 09:56:26.640112 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.640118 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.640124 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.640130 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.640136 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.640142 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.640148 | controller |       -- \'
2026-03-14 09:56:26.640153 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.640159 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.640165 | controller |     - '    logger.go:42: 09:38:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.640171 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.640186 | controller |     - '    logger.go:42: 09:38:01 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.640192 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.640198 | controller |       found'
2026-03-14 09:56:26.640204 | controller |     - '    logger.go:42: 09:38:02 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.640210 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.640219 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.640225 | controller |     - '        '
2026-03-14 09:56:26.640231 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.640241 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.640247 | controller |     - '        '
2026-03-14 09:56:26.640253 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.640259 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.640265 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.640271 | controller |     - '          exit 1'
2026-03-14 09:56:26.640279 | controller |     - '        fi'
2026-03-14 09:56:26.640285 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.640291 | controller |     - '        '
2026-03-14 09:56:26.640297 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.640303 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.640309 | controller |       | base64 -d)'
2026-03-14 09:56:26.640315 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.640321 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.640327 | controller |     - '          exit 1'
2026-03-14 09:56:26.640333 | controller |     - '        fi'
2026-03-14 09:56:26.640339 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.640345 | controller |     - '        '
2026-03-14 09:56:26.640351 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.640357 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.640362 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.640368 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.640374 | controller |       --timeout=300s'
2026-03-14 09:56:26.640380 | controller |     - '        '
2026-03-14 09:56:26.640386 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.640392 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.640398 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.640404 | controller |     - '        '
2026-03-14 09:56:26.640410 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.640416 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.640422 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.640428 | controller |     - '        '
2026-03-14 09:56:26.640434 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.640440 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.640446 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.640451 | controller |       available"'
2026-03-14 09:56:26.640457 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.640463 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.640469 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.640474 | controller |     - '          exit 1'
2026-03-14 09:56:26.640480 | controller |     - '        fi'
2026-03-14 09:56:26.640487 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.640493 | controller |     - '        '
2026-03-14 09:56:26.640498 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.640504 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.640515 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.640521 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.640527 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.640533 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.640542 | controller |       -- \'
2026-03-14 09:56:26.640547 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.640554 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.640560 | controller |     - '    logger.go:42: 09:38:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.640566 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.640581 | controller |     - '    logger.go:42: 09:38:02 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.640587 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.640606 | controller |       found'
2026-03-14 09:56:26.640615 | controller |     - '    logger.go:42: 09:38:03 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.640622 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.640628 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.640634 | controller |     - '        '
2026-03-14 09:56:26.640640 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.640645 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.640651 | controller |     - '        '
2026-03-14 09:56:26.640657 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.640663 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.640669 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.640675 | controller |     - '          exit 1'
2026-03-14 09:56:26.640681 | controller |     - '        fi'
2026-03-14 09:56:26.640687 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.640692 | controller |     - '        '
2026-03-14 09:56:26.640698 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.640704 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.640710 | controller |       | base64 -d)'
2026-03-14 09:56:26.640716 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.640722 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.640728 | controller |     - '          exit 1'
2026-03-14 09:56:26.640734 | controller |     - '        fi'
2026-03-14 09:56:26.640739 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.640745 | controller |     - '        '
2026-03-14 09:56:26.640751 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.640757 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.640763 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.640769 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.640775 | controller |       --timeout=300s'
2026-03-14 09:56:26.640781 | controller |     - '        '
2026-03-14 09:56:26.640787 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.640793 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.640799 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.640808 | controller |     - '        '
2026-03-14 09:56:26.640814 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.640820 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.640825 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.640831 | controller |     - '        '
2026-03-14 09:56:26.640837 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.640843 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.640849 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.640855 | controller |       available"'
2026-03-14 09:56:26.640861 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.640867 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.640873 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.640879 | controller |     - '          exit 1'
2026-03-14 09:56:26.640885 | controller |     - '        fi'
2026-03-14 09:56:26.640891 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.640897 | controller |     - '        '
2026-03-14 09:56:26.640903 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.640909 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.640915 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.640921 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.640926 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.640932 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.640938 | controller |       -- \'
2026-03-14 09:56:26.640944 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.640950 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.640956 | controller |     - '    logger.go:42: 09:38:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.640962 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.640968 | controller |     - '    logger.go:42: 09:38:03 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.640984 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.640991 | controller |       found'
2026-03-14 09:56:26.641001 | controller |     - '    logger.go:42: 09:38:04 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.641007 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.641013 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.641019 | controller |     - '        '
2026-03-14 09:56:26.641026 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.641032 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.641042 | controller |     - '        '
2026-03-14 09:56:26.641049 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.641055 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.641062 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.641068 | controller |     - '          exit 1'
2026-03-14 09:56:26.641074 | controller |     - '        fi'
2026-03-14 09:56:26.641083 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.641090 | controller |     - '        '
2026-03-14 09:56:26.641096 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.641102 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.641108 | controller |       | base64 -d)'
2026-03-14 09:56:26.641115 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.641121 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.641128 | controller |     - '          exit 1'
2026-03-14 09:56:26.641134 | controller |     - '        fi'
2026-03-14 09:56:26.641140 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.641146 | controller |     - '        '
2026-03-14 09:56:26.641152 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.641158 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.641164 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.641170 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.641176 | controller |       --timeout=300s'
2026-03-14 09:56:26.641182 | controller |     - '        '
2026-03-14 09:56:26.641188 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.641194 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.641200 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.641206 | controller |     - '        '
2026-03-14 09:56:26.641212 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.641218 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.641224 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.641234 | controller |     - '        '
2026-03-14 09:56:26.641240 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.641246 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.641252 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.641258 | controller |       available"'
2026-03-14 09:56:26.641263 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.641269 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.641275 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.641281 | controller |     - '          exit 1'
2026-03-14 09:56:26.641287 | controller |     - '        fi'
2026-03-14 09:56:26.641293 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.641299 | controller |     - '        '
2026-03-14 09:56:26.641305 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.641310 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.641316 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.641322 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.641328 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.641334 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.641343 | controller |       -- \'
2026-03-14 09:56:26.641349 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.641355 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.641360 | controller |     - '    logger.go:42: 09:38:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.641367 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.641373 | controller |     - '    logger.go:42: 09:38:04 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.641390 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.641397 | controller |       found'
2026-03-14 09:56:26.641403 | controller |     - '    logger.go:42: 09:38:05 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.641409 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.641415 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.641421 | controller |     - '        '
2026-03-14 09:56:26.641427 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.641432 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.641438 | controller |     - '        '
2026-03-14 09:56:26.641444 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.641449 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.641455 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.641461 | controller |     - '          exit 1'
2026-03-14 09:56:26.641467 | controller |     - '        fi'
2026-03-14 09:56:26.641473 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.641479 | controller |     - '        '
2026-03-14 09:56:26.641485 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.641490 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.641496 | controller |       | base64 -d)'
2026-03-14 09:56:26.641502 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.641508 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.641514 | controller |     - '          exit 1'
2026-03-14 09:56:26.641519 | controller |     - '        fi'
2026-03-14 09:56:26.641525 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.641531 | controller |     - '        '
2026-03-14 09:56:26.641536 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.641542 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.641548 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.641554 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.641560 | controller |       --timeout=300s'
2026-03-14 09:56:26.641566 | controller |     - '        '
2026-03-14 09:56:26.641572 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.641578 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.641584 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.641590 | controller |     - '        '
2026-03-14 09:56:26.641615 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.641624 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.641630 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.641636 | controller |     - '        '
2026-03-14 09:56:26.641646 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.641652 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.641657 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.641663 | controller |       available"'
2026-03-14 09:56:26.641669 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.641674 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.641680 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.641686 | controller |     - '          exit 1'
2026-03-14 09:56:26.641695 | controller |     - '        fi'
2026-03-14 09:56:26.641701 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.641707 | controller |     - '        '
2026-03-14 09:56:26.641713 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.641719 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.641740 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.641746 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.641752 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.641758 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.641764 | controller |       -- \'
2026-03-14 09:56:26.641770 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.641776 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.641782 | controller |     - '    logger.go:42: 09:38:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.641788 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.641794 | controller |     - '    logger.go:42: 09:38:05 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.641799 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.641805 | controller |       found'
2026-03-14 09:56:26.641823 | controller |     - '    logger.go:42: 09:38:06 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.641830 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.641836 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.641842 | controller |     - '        '
2026-03-14 09:56:26.641848 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.641854 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.641860 | controller |     - '        '
2026-03-14 09:56:26.641866 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.641875 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.641881 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.641887 | controller |     - '          exit 1'
2026-03-14 09:56:26.641893 | controller |     - '        fi'
2026-03-14 09:56:26.641898 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.641904 | controller |     - '        '
2026-03-14 09:56:26.641910 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.641918 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.641924 | controller |       | base64 -d)'
2026-03-14 09:56:26.641930 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.641936 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.641944 | controller |     - '          exit 1'
2026-03-14 09:56:26.641950 | controller |     - '        fi'
2026-03-14 09:56:26.641956 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.641961 | controller |     - '        '
2026-03-14 09:56:26.641967 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.641973 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.641979 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.641985 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.641991 | controller |       --timeout=300s'
2026-03-14 09:56:26.641997 | controller |     - '        '
2026-03-14 09:56:26.642002 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.642008 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.642014 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.642020 | controller |     - '        '
2026-03-14 09:56:26.642026 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.642032 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.642038 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.642044 | controller |     - '        '
2026-03-14 09:56:26.642050 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.642056 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.642062 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.642068 | controller |       available"'
2026-03-14 09:56:26.642074 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.642080 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.642086 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.642091 | controller |     - '          exit 1'
2026-03-14 09:56:26.642097 | controller |     - '        fi'
2026-03-14 09:56:26.642103 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.642109 | controller |     - '        '
2026-03-14 09:56:26.642114 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.642120 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.642126 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.642131 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.642137 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.642143 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.642149 | controller |       -- \'
2026-03-14 09:56:26.642156 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.642162 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.642168 | controller |     - '    logger.go:42: 09:38:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.642174 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.642179 | controller |     - '    logger.go:42: 09:38:07 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.642188 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.642194 | controller |       found'
2026-03-14 09:56:26.642201 | controller |     - '    logger.go:42: 09:38:08 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.642218 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.642224 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.642230 | controller |     - '        '
2026-03-14 09:56:26.642236 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.642242 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.642248 | controller |     - '        '
2026-03-14 09:56:26.642254 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.642259 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.642266 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.642271 | controller |     - '          exit 1'
2026-03-14 09:56:26.642278 | controller |     - '        fi'
2026-03-14 09:56:26.642284 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.642289 | controller |     - '        '
2026-03-14 09:56:26.642295 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.642301 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.642307 | controller |       | base64 -d)'
2026-03-14 09:56:26.642314 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.642320 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.642325 | controller |     - '          exit 1'
2026-03-14 09:56:26.642330 | controller |     - '        fi'
2026-03-14 09:56:26.642336 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.642341 | controller |     - '        '
2026-03-14 09:56:26.642346 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.642351 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.642356 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.642362 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.642367 | controller |       --timeout=300s'
2026-03-14 09:56:26.642372 | controller |     - '        '
2026-03-14 09:56:26.642377 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.642388 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.642393 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.642398 | controller |     - '        '
2026-03-14 09:56:26.642403 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.642408 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.642414 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.642419 | controller |     - '        '
2026-03-14 09:56:26.642424 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.642429 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.642435 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.642440 | controller |       available"'
2026-03-14 09:56:26.642445 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.642450 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.642458 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.642464 | controller |     - '          exit 1'
2026-03-14 09:56:26.642469 | controller |     - '        fi'
2026-03-14 09:56:26.642474 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.642479 | controller |     - '        '
2026-03-14 09:56:26.642484 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.642490 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.642495 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.642500 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.642505 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.642511 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.642516 | controller |       -- \'
2026-03-14 09:56:26.642521 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.642526 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.642532 | controller |     - '    logger.go:42: 09:38:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.642537 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.642542 | controller |     - '    logger.go:42: 09:38:08 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.642547 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.642552 | controller |       found'
2026-03-14 09:56:26.642558 | controller |     - '    logger.go:42: 09:38:09 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.642563 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.642577 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.642583 | controller |     - '        '
2026-03-14 09:56:26.642588 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.642607 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.642616 | controller |     - '        '
2026-03-14 09:56:26.642621 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.642626 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.642631 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.642637 | controller |     - '          exit 1'
2026-03-14 09:56:26.642642 | controller |     - '        fi'
2026-03-14 09:56:26.642647 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.642652 | controller |     - '        '
2026-03-14 09:56:26.642658 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.642663 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.642668 | controller |       | base64 -d)'
2026-03-14 09:56:26.642674 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.642679 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.642684 | controller |     - '          exit 1'
2026-03-14 09:56:26.642689 | controller |     - '        fi'
2026-03-14 09:56:26.642694 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.642700 | controller |     - '        '
2026-03-14 09:56:26.642705 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.642710 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.642718 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.642726 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.642731 | controller |       --timeout=300s'
2026-03-14 09:56:26.642736 | controller |     - '        '
2026-03-14 09:56:26.642742 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.642747 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.642754 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.642760 | controller |     - '        '
2026-03-14 09:56:26.642765 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.642770 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.642775 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.642780 | controller |     - '        '
2026-03-14 09:56:26.642786 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.642791 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.642796 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.642801 | controller |       available"'
2026-03-14 09:56:26.642806 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.642812 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.642817 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.642822 | controller |     - '          exit 1'
2026-03-14 09:56:26.642827 | controller |     - '        fi'
2026-03-14 09:56:26.642832 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.642838 | controller |     - '        '
2026-03-14 09:56:26.642843 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.642848 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.642853 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.642858 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.642864 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.642869 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.642874 | controller |       -- \'
2026-03-14 09:56:26.642879 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.642885 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.642890 | controller |     - '    logger.go:42: 09:38:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.642895 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.642900 | controller |     - '    logger.go:42: 09:38:09 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.642906 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.642911 | controller |       found'
2026-03-14 09:56:26.642918 | controller |     - '    logger.go:42: 09:38:10 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.642923 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.642929 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.642934 | controller |     - '        '
2026-03-14 09:56:26.642948 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.642965 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.642973 | controller |     - '        '
2026-03-14 09:56:26.642978 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.642984 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.642989 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.642994 | controller |     - '          exit 1'
2026-03-14 09:56:26.642999 | controller |     - '        fi'
2026-03-14 09:56:26.643005 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.643010 | controller |     - '        '
2026-03-14 09:56:26.643015 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.643020 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.643025 | controller |       | base64 -d)'
2026-03-14 09:56:26.643030 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.643036 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.643041 | controller |     - '          exit 1'
2026-03-14 09:56:26.643046 | controller |     - '        fi'
2026-03-14 09:56:26.643051 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.643057 | controller |     - '        '
2026-03-14 09:56:26.643062 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.643067 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.643073 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.643078 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.643083 | controller |       --timeout=300s'
2026-03-14 09:56:26.643091 | controller |     - '        '
2026-03-14 09:56:26.643097 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.643102 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.643107 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.643112 | controller |     - '        '
2026-03-14 09:56:26.643118 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.643123 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.643128 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.643133 | controller |     - '        '
2026-03-14 09:56:26.643139 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.643144 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.643149 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.643154 | controller |       available"'
2026-03-14 09:56:26.643159 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.643165 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.643170 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.643175 | controller |     - '          exit 1'
2026-03-14 09:56:26.643180 | controller |     - '        fi'
2026-03-14 09:56:26.643186 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.643191 | controller |     - '        '
2026-03-14 09:56:26.643196 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.643201 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.643209 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.643214 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.643219 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.643224 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.643229 | controller |       -- \'
2026-03-14 09:56:26.643235 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.643240 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.643245 | controller |     - '    logger.go:42: 09:38:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.643250 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.643255 | controller |     - '    logger.go:42: 09:38:10 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.643261 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.643266 | controller |       found'
2026-03-14 09:56:26.643272 | controller |     - '    logger.go:42: 09:38:11 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.643278 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.643284 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.643290 | controller |     - '        '
2026-03-14 09:56:26.643304 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.643310 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.643316 | controller |     - '        '
2026-03-14 09:56:26.643322 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.643328 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.643335 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.643342 | controller |     - '          exit 1'
2026-03-14 09:56:26.643348 | controller |     - '        fi'
2026-03-14 09:56:26.643355 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.643362 | controller |     - '        '
2026-03-14 09:56:26.643368 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.643374 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.643380 | controller |       | base64 -d)'
2026-03-14 09:56:26.643385 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.643391 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.643397 | controller |     - '          exit 1'
2026-03-14 09:56:26.643403 | controller |     - '        fi'
2026-03-14 09:56:26.643409 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.643415 | controller |     - '        '
2026-03-14 09:56:26.643420 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.643426 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.643432 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.643441 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.643447 | controller |       --timeout=300s'
2026-03-14 09:56:26.643452 | controller |     - '        '
2026-03-14 09:56:26.643458 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.643464 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.643470 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.643481 | controller |     - '        '
2026-03-14 09:56:26.643487 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.643492 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.643498 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.643504 | controller |     - '        '
2026-03-14 09:56:26.643510 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.643516 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.643521 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.643527 | controller |       available"'
2026-03-14 09:56:26.643533 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.643539 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.643547 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.643553 | controller |     - '          exit 1'
2026-03-14 09:56:26.643559 | controller |     - '        fi'
2026-03-14 09:56:26.643565 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.643570 | controller |     - '        '
2026-03-14 09:56:26.643576 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.643582 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.643588 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.643610 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.643620 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.643626 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.643631 | controller |       -- \'
2026-03-14 09:56:26.643637 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.643643 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.643649 | controller |     - '    logger.go:42: 09:38:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.643655 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.643661 | controller |     - '    logger.go:42: 09:38:11 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.643667 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.643672 | controller |       found'
2026-03-14 09:56:26.643678 | controller |     - '    logger.go:42: 09:38:12 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.643684 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.643690 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.643696 | controller |     - '        '
2026-03-14 09:56:26.643702 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.643715 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.643721 | controller |     - '        '
2026-03-14 09:56:26.643727 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.643733 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.643739 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.643745 | controller |     - '          exit 1'
2026-03-14 09:56:26.643751 | controller |     - '        fi'
2026-03-14 09:56:26.643757 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.643765 | controller |     - '        '
2026-03-14 09:56:26.643771 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.643777 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.643783 | controller |       | base64 -d)'
2026-03-14 09:56:26.643789 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.643795 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.643801 | controller |     - '          exit 1'
2026-03-14 09:56:26.643807 | controller |     - '        fi'
2026-03-14 09:56:26.643813 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.643819 | controller |     - '        '
2026-03-14 09:56:26.643825 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.643831 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.643837 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.643843 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.643848 | controller |       --timeout=300s'
2026-03-14 09:56:26.643854 | controller |     - '        '
2026-03-14 09:56:26.643860 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.643866 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.643872 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.643878 | controller |     - '        '
2026-03-14 09:56:26.643884 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.643890 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.643896 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.643902 | controller |     - '        '
2026-03-14 09:56:26.643908 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.643914 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.643920 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.643926 | controller |       available"'
2026-03-14 09:56:26.643932 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.643938 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.643944 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.643950 | controller |     - '          exit 1'
2026-03-14 09:56:26.643956 | controller |     - '        fi'
2026-03-14 09:56:26.643962 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.643968 | controller |     - '        '
2026-03-14 09:56:26.643974 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.643980 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.643986 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.643995 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.644001 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.644007 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.644013 | controller |       -- \'
2026-03-14 09:56:26.644019 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.644028 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.644034 | controller |     - '    logger.go:42: 09:38:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.644040 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.644046 | controller |     - '    logger.go:42: 09:38:12 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.644052 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.644058 | controller |       found'
2026-03-14 09:56:26.644065 | controller |     - '    logger.go:42: 09:38:13 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.644071 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.644077 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.644083 | controller |     - '        '
2026-03-14 09:56:26.644089 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.644095 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.644110 | controller |     - '        '
2026-03-14 09:56:26.644116 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.644122 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.644129 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.644135 | controller |     - '          exit 1'
2026-03-14 09:56:26.644141 | controller |     - '        fi'
2026-03-14 09:56:26.644147 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.644155 | controller |     - '        '
2026-03-14 09:56:26.644161 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.644168 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.644178 | controller |       | base64 -d)'
2026-03-14 09:56:26.644184 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.644190 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.644196 | controller |     - '          exit 1'
2026-03-14 09:56:26.644202 | controller |     - '        fi'
2026-03-14 09:56:26.644208 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.644214 | controller |     - '        '
2026-03-14 09:56:26.644220 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.644226 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.644231 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.644237 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.644246 | controller |       --timeout=300s'
2026-03-14 09:56:26.644251 | controller |     - '        '
2026-03-14 09:56:26.644257 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.644262 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.644268 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.644273 | controller |     - '        '
2026-03-14 09:56:26.644279 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.644284 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.644290 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.644295 | controller |     - '        '
2026-03-14 09:56:26.644301 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.644309 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.644314 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.644320 | controller |       available"'
2026-03-14 09:56:26.644326 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.644332 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.644338 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.644345 | controller |     - '          exit 1'
2026-03-14 09:56:26.644351 | controller |     - '        fi'
2026-03-14 09:56:26.644357 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.644363 | controller |     - '        '
2026-03-14 09:56:26.644370 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.644376 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.644382 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.644389 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.644395 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.644401 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.644407 | controller |       -- \'
2026-03-14 09:56:26.644413 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.644420 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.644426 | controller |     - '    logger.go:42: 09:38:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.644432 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.644439 | controller |     - '    logger.go:42: 09:38:14 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.644445 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.644451 | controller |       found'
2026-03-14 09:56:26.644457 | controller |     - '    logger.go:42: 09:38:15 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.644464 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.644470 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.644476 | controller |     - '        '
2026-03-14 09:56:26.644482 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.644489 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.644495 | controller |     - '        '
2026-03-14 09:56:26.644512 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.644519 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.644526 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.644532 | controller |     - '          exit 1'
2026-03-14 09:56:26.644539 | controller |     - '        fi'
2026-03-14 09:56:26.644545 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.644551 | controller |     - '        '
2026-03-14 09:56:26.644561 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.644568 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.644574 | controller |       | base64 -d)'
2026-03-14 09:56:26.644581 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.644587 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.644607 | controller |     - '          exit 1'
2026-03-14 09:56:26.644621 | controller |     - '        fi'
2026-03-14 09:56:26.644628 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.644635 | controller |     - '        '
2026-03-14 09:56:26.644641 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.644647 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.644654 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.644660 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.644666 | controller |       --timeout=300s'
2026-03-14 09:56:26.644673 | controller |     - '        '
2026-03-14 09:56:26.644679 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.644685 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.644691 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.644697 | controller |     - '        '
2026-03-14 09:56:26.644704 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.644710 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.644716 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.644722 | controller |     - '        '
2026-03-14 09:56:26.644728 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.644735 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.644741 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.644747 | controller |       available"'
2026-03-14 09:56:26.644753 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.644759 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.644765 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.644772 | controller |     - '          exit 1'
2026-03-14 09:56:26.644778 | controller |     - '        fi'
2026-03-14 09:56:26.644784 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.644790 | controller |     - '        '
2026-03-14 09:56:26.644796 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.644803 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.644809 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.644815 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.644821 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.644827 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.644834 | controller |       -- \'
2026-03-14 09:56:26.644840 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.644850 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.644856 | controller |     - '    logger.go:42: 09:38:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.644862 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.644869 | controller |     - '    logger.go:42: 09:38:15 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.644875 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.644884 | controller |       found'
2026-03-14 09:56:26.644890 | controller |     - '    logger.go:42: 09:38:16 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.644896 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.644903 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.644909 | controller |     - '        '
2026-03-14 09:56:26.644915 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.644921 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.644927 | controller |     - '        '
2026-03-14 09:56:26.644945 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.644952 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.644962 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.644969 | controller |     - '          exit 1'
2026-03-14 09:56:26.644975 | controller |     - '        fi'
2026-03-14 09:56:26.644982 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.644988 | controller |     - '        '
2026-03-14 09:56:26.644994 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.645000 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.645007 | controller |       | base64 -d)'
2026-03-14 09:56:26.645013 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.645019 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.645025 | controller |     - '          exit 1'
2026-03-14 09:56:26.645032 | controller |     - '        fi'
2026-03-14 09:56:26.645038 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.645044 | controller |     - '        '
2026-03-14 09:56:26.645050 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.645057 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.645063 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.645069 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.645075 | controller |       --timeout=300s'
2026-03-14 09:56:26.645081 | controller |     - '        '
2026-03-14 09:56:26.645088 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.645094 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.645100 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.645106 | controller |     - '        '
2026-03-14 09:56:26.645113 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.645119 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.645125 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.645131 | controller |     - '        '
2026-03-14 09:56:26.645138 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.645147 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.645154 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.645160 | controller |       available"'
2026-03-14 09:56:26.645166 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.645172 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.645178 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.645188 | controller |     - '          exit 1'
2026-03-14 09:56:26.645194 | controller |     - '        fi'
2026-03-14 09:56:26.645201 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.645207 | controller |     - '        '
2026-03-14 09:56:26.645213 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.645220 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.645226 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.645232 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.645239 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.645245 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.645251 | controller |       -- \'
2026-03-14 09:56:26.645257 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.645264 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.645270 | controller |     - '    logger.go:42: 09:38:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.645277 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.645283 | controller |     - '    logger.go:42: 09:38:16 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.645289 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.645296 | controller |       found'
2026-03-14 09:56:26.645302 | controller |     - '    logger.go:42: 09:38:17 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.645308 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.645315 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.645321 | controller |     - '        '
2026-03-14 09:56:26.645328 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.645334 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.645340 | controller |     - '        '
2026-03-14 09:56:26.645347 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.645353 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.645371 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.645383 | controller |     - '          exit 1'
2026-03-14 09:56:26.645389 | controller |     - '        fi'
2026-03-14 09:56:26.645395 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.645401 | controller |     - '        '
2026-03-14 09:56:26.645408 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.645414 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.645421 | controller |       | base64 -d)'
2026-03-14 09:56:26.645427 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.645433 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.645439 | controller |     - '          exit 1'
2026-03-14 09:56:26.645444 | controller |     - '        fi'
2026-03-14 09:56:26.645449 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.645455 | controller |     - '        '
2026-03-14 09:56:26.645462 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.645468 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.645474 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.645480 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.645490 | controller |       --timeout=300s'
2026-03-14 09:56:26.645496 | controller |     - '        '
2026-03-14 09:56:26.645503 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.645509 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.645515 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.645526 | controller |     - '        '
2026-03-14 09:56:26.645532 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.645539 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.645545 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.645551 | controller |     - '        '
2026-03-14 09:56:26.645558 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.645564 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.645570 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.645577 | controller |       available"'
2026-03-14 09:56:26.645583 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.645590 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.645616 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.645623 | controller |     - '          exit 1'
2026-03-14 09:56:26.645630 | controller |     - '        fi'
2026-03-14 09:56:26.645636 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.645643 | controller |     - '        '
2026-03-14 09:56:26.645649 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.645655 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.645661 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.645667 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.645674 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.645680 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.645686 | controller |       -- \'
2026-03-14 09:56:26.645692 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.645698 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.645705 | controller |     - '    logger.go:42: 09:38:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.645711 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.645717 | controller |     - '    logger.go:42: 09:38:17 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.645724 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.645730 | controller |       found'
2026-03-14 09:56:26.645736 | controller |     - '    logger.go:42: 09:38:18 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.645742 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.645748 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.645755 | controller |     - '        '
2026-03-14 09:56:26.645761 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.645767 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.645777 | controller |     - '        '
2026-03-14 09:56:26.645783 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.645790 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.645807 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.645816 | controller |     - '          exit 1'
2026-03-14 09:56:26.645822 | controller |     - '        fi'
2026-03-14 09:56:26.645829 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.645835 | controller |     - '        '
2026-03-14 09:56:26.645841 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.645847 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.645853 | controller |       | base64 -d)'
2026-03-14 09:56:26.645860 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.645866 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.645872 | controller |     - '          exit 1'
2026-03-14 09:56:26.645879 | controller |     - '        fi'
2026-03-14 09:56:26.645885 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.645892 | controller |     - '        '
2026-03-14 09:56:26.645898 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.645904 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.645910 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.645916 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.645923 | controller |       --timeout=300s'
2026-03-14 09:56:26.645929 | controller |     - '        '
2026-03-14 09:56:26.645935 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.645942 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.645948 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.645954 | controller |     - '        '
2026-03-14 09:56:26.645961 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.645967 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.645973 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.645979 | controller |     - '        '
2026-03-14 09:56:26.645986 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.645992 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.645998 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.646009 | controller |       available"'
2026-03-14 09:56:26.646016 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.646022 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.646028 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.646034 | controller |     - '          exit 1'
2026-03-14 09:56:26.646041 | controller |     - '        fi'
2026-03-14 09:56:26.646047 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.646053 | controller |     - '        '
2026-03-14 09:56:26.646065 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.646071 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.646077 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.646086 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.646093 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.646099 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.646108 | controller |       -- \'
2026-03-14 09:56:26.646114 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.646120 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.646126 | controller |     - '    logger.go:42: 09:38:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.646132 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.646139 | controller |     - '    logger.go:42: 09:38:18 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.646145 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.646151 | controller |       found'
2026-03-14 09:56:26.646157 | controller |     - '    logger.go:42: 09:38:19 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.646163 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.646169 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.646175 | controller |     - '        '
2026-03-14 09:56:26.646181 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.646187 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.646193 | controller |     - '        '
2026-03-14 09:56:26.646200 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.646206 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.646212 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.646218 | controller |     - '          exit 1'
2026-03-14 09:56:26.646235 | controller |     - '        fi'
2026-03-14 09:56:26.646243 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.646249 | controller |     - '        '
2026-03-14 09:56:26.646255 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.646261 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.646267 | controller |       | base64 -d)'
2026-03-14 09:56:26.646273 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.646279 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.646286 | controller |     - '          exit 1'
2026-03-14 09:56:26.646292 | controller |     - '        fi'
2026-03-14 09:56:26.646298 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.646304 | controller |     - '        '
2026-03-14 09:56:26.646310 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.646316 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.646322 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.646328 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.646335 | controller |       --timeout=300s'
2026-03-14 09:56:26.646341 | controller |     - '        '
2026-03-14 09:56:26.646347 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.646353 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.646359 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.646365 | controller |     - '        '
2026-03-14 09:56:26.646375 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.646381 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.646387 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.646394 | controller |     - '        '
2026-03-14 09:56:26.646400 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.646406 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.646412 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.646418 | controller |       available"'
2026-03-14 09:56:26.646424 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.646430 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.646436 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.646442 | controller |     - '          exit 1'
2026-03-14 09:56:26.646448 | controller |     - '        fi'
2026-03-14 09:56:26.646454 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.646459 | controller |     - '        '
2026-03-14 09:56:26.646464 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.646470 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.646476 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.646482 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.646488 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.646495 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.646501 | controller |       -- \'
2026-03-14 09:56:26.646507 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.646513 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.646519 | controller |     - '    logger.go:42: 09:38:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.646525 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.646532 | controller |     - '    logger.go:42: 09:38:20 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.646538 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.646544 | controller |       found'
2026-03-14 09:56:26.646550 | controller |     - '    logger.go:42: 09:38:21 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.646556 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.646562 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.646568 | controller |     - '        '
2026-03-14 09:56:26.646574 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.646580 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.646587 | controller |     - '        '
2026-03-14 09:56:26.646607 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.646619 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.646626 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.646632 | controller |     - '          exit 1'
2026-03-14 09:56:26.646638 | controller |     - '        fi'
2026-03-14 09:56:26.646655 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.646662 | controller |     - '        '
2026-03-14 09:56:26.646672 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.646679 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.646685 | controller |       | base64 -d)'
2026-03-14 09:56:26.646691 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.646697 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.646704 | controller |     - '          exit 1'
2026-03-14 09:56:26.646710 | controller |     - '        fi'
2026-03-14 09:56:26.646716 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.646722 | controller |     - '        '
2026-03-14 09:56:26.646728 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.646734 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.646740 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.646746 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.646752 | controller |       --timeout=300s'
2026-03-14 09:56:26.646759 | controller |     - '        '
2026-03-14 09:56:26.646765 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.646771 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.646777 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.646783 | controller |     - '        '
2026-03-14 09:56:26.646789 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.646795 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.646801 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.646807 | controller |     - '        '
2026-03-14 09:56:26.646814 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.646820 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.646826 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.646832 | controller |       available"'
2026-03-14 09:56:26.646838 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.646844 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.646850 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.646857 | controller |     - '          exit 1'
2026-03-14 09:56:26.646863 | controller |     - '        fi'
2026-03-14 09:56:26.646869 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.646880 | controller |     - '        '
2026-03-14 09:56:26.646886 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.646892 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.646898 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.646904 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.646910 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.646919 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.646925 | controller |       -- \'
2026-03-14 09:56:26.646932 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.646941 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.646947 | controller |     - '    logger.go:42: 09:38:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.647012 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.647021 | controller |     - '    logger.go:42: 09:38:21 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.647027 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.647033 | controller |       found'
2026-03-14 09:56:26.647039 | controller |     - '    logger.go:42: 09:38:22 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.647045 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.647051 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.647058 | controller |     - '        '
2026-03-14 09:56:26.647064 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.647070 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.647076 | controller |     - '        '
2026-03-14 09:56:26.647082 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.647088 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.647094 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.647100 | controller |     - '          exit 1'
2026-03-14 09:56:26.647107 | controller |     - '        fi'
2026-03-14 09:56:26.647113 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.647119 | controller |     - '        '
2026-03-14 09:56:26.647136 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.647143 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.647150 | controller |       | base64 -d)'
2026-03-14 09:56:26.647156 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.647162 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.647169 | controller |     - '          exit 1'
2026-03-14 09:56:26.647175 | controller |     - '        fi'
2026-03-14 09:56:26.647181 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.647187 | controller |     - '        '
2026-03-14 09:56:26.647193 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.647200 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.647206 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.647212 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.647218 | controller |       --timeout=300s'
2026-03-14 09:56:26.647224 | controller |     - '        '
2026-03-14 09:56:26.647230 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.647236 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.647242 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.647248 | controller |     - '        '
2026-03-14 09:56:26.647255 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.647261 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.647267 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.647273 | controller |     - '        '
2026-03-14 09:56:26.647279 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.647289 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.647295 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.647302 | controller |       available"'
2026-03-14 09:56:26.647308 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.647314 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.647324 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.647330 | controller |     - '          exit 1'
2026-03-14 09:56:26.647336 | controller |     - '        fi'
2026-03-14 09:56:26.647342 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.647348 | controller |     - '        '
2026-03-14 09:56:26.647353 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.647359 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.647365 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.647371 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.647377 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.647384 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.647389 | controller |       -- \'
2026-03-14 09:56:26.647397 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.647404 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.647411 | controller |     - '    logger.go:42: 09:38:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.647418 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.647425 | controller |     - '    logger.go:42: 09:38:22 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.647431 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.647437 | controller |       found'
2026-03-14 09:56:26.647443 | controller |     - '    logger.go:42: 09:38:23 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.647450 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.647456 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.647462 | controller |     - '        '
2026-03-14 09:56:26.647468 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.647474 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.647480 | controller |     - '        '
2026-03-14 09:56:26.647486 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.647492 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.647498 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.647505 | controller |     - '          exit 1'
2026-03-14 09:56:26.647511 | controller |     - '        fi'
2026-03-14 09:56:26.647517 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.647523 | controller |     - '        '
2026-03-14 09:56:26.647529 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.647546 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.647553 | controller |       | base64 -d)'
2026-03-14 09:56:26.647559 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.647566 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.647572 | controller |     - '          exit 1'
2026-03-14 09:56:26.647578 | controller |     - '        fi'
2026-03-14 09:56:26.647588 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.647609 | controller |     - '        '
2026-03-14 09:56:26.647619 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.647626 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.647632 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.647638 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.647644 | controller |       --timeout=300s'
2026-03-14 09:56:26.647650 | controller |     - '        '
2026-03-14 09:56:26.647656 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.647662 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.647682 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.647689 | controller |     - '        '
2026-03-14 09:56:26.647695 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.647701 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.647707 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.647713 | controller |     - '        '
2026-03-14 09:56:26.647719 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.647726 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.647732 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.647738 | controller |       available"'
2026-03-14 09:56:26.647744 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.647750 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.647756 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.647762 | controller |     - '          exit 1'
2026-03-14 09:56:26.647769 | controller |     - '        fi'
2026-03-14 09:56:26.647775 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.647781 | controller |     - '        '
2026-03-14 09:56:26.647787 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.647797 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.647803 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.647809 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.647815 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.647822 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.647828 | controller |       -- \'
2026-03-14 09:56:26.647834 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.647840 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.647846 | controller |     - '    logger.go:42: 09:38:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.647853 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.647859 | controller |     - '    logger.go:42: 09:38:23 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.647865 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.647875 | controller |       found'
2026-03-14 09:56:26.647881 | controller |     - '    logger.go:42: 09:38:24 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.647887 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.647893 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.647899 | controller |     - '        '
2026-03-14 09:56:26.647906 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.647911 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.647917 | controller |     - '        '
2026-03-14 09:56:26.647923 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.647929 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.647935 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.647942 | controller |     - '          exit 1'
2026-03-14 09:56:26.647948 | controller |     - '        fi'
2026-03-14 09:56:26.647954 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.647960 | controller |     - '        '
2026-03-14 09:56:26.647966 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.647984 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.647991 | controller |       | base64 -d)'
2026-03-14 09:56:26.647997 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.648003 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.648009 | controller |     - '          exit 1'
2026-03-14 09:56:26.648016 | controller |     - '        fi'
2026-03-14 09:56:26.648022 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.648028 | controller |     - '        '
2026-03-14 09:56:26.648034 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.648040 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.648046 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.648052 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.648058 | controller |       --timeout=300s'
2026-03-14 09:56:26.648064 | controller |     - '        '
2026-03-14 09:56:26.648071 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.648081 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.648088 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.648094 | controller |     - '        '
2026-03-14 09:56:26.648100 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.648106 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.648112 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.648118 | controller |     - '        '
2026-03-14 09:56:26.648124 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.648130 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.648137 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.648143 | controller |       available"'
2026-03-14 09:56:26.648149 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.648155 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.648161 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.648167 | controller |     - '          exit 1'
2026-03-14 09:56:26.648179 | controller |     - '        fi'
2026-03-14 09:56:26.648185 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.648191 | controller |     - '        '
2026-03-14 09:56:26.648197 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.648204 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.648210 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.648216 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.648222 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.648228 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.648235 | controller |       -- \'
2026-03-14 09:56:26.648241 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.648246 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.648251 | controller |     - '    logger.go:42: 09:38:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.648257 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.648263 | controller |     - '    logger.go:42: 09:38:24 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.648269 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.648275 | controller |       found'
2026-03-14 09:56:26.648281 | controller |     - '    logger.go:42: 09:38:25 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.648288 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.648294 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.648300 | controller |     - '        '
2026-03-14 09:56:26.648306 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.648312 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.648318 | controller |     - '        '
2026-03-14 09:56:26.648324 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.648331 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.648337 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.648343 | controller |     - '          exit 1'
2026-03-14 09:56:26.648349 | controller |     - '        fi'
2026-03-14 09:56:26.648355 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.648361 | controller |     - '        '
2026-03-14 09:56:26.648367 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.648373 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.648390 | controller |       | base64 -d)'
2026-03-14 09:56:26.648397 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.648403 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.648409 | controller |     - '          exit 1'
2026-03-14 09:56:26.648415 | controller |     - '        fi'
2026-03-14 09:56:26.648422 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.648427 | controller |     - '        '
2026-03-14 09:56:26.648433 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.648439 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.648445 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.648451 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.648457 | controller |       --timeout=300s'
2026-03-14 09:56:26.648467 | controller |     - '        '
2026-03-14 09:56:26.648473 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.648480 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.648486 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.648492 | controller |     - '        '
2026-03-14 09:56:26.648498 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.648504 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.648514 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.648520 | controller |     - '        '
2026-03-14 09:56:26.648526 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.648533 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.648539 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.648545 | controller |       available"'
2026-03-14 09:56:26.648551 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.648557 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.648563 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.648569 | controller |     - '          exit 1'
2026-03-14 09:56:26.648575 | controller |     - '        fi'
2026-03-14 09:56:26.648581 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.648587 | controller |     - '        '
2026-03-14 09:56:26.648610 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.648620 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.648627 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.648633 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.648642 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.648649 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.648655 | controller |       -- \'
2026-03-14 09:56:26.648661 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.648667 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.648673 | controller |     - '    logger.go:42: 09:38:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.648679 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.648686 | controller |     - '    logger.go:42: 09:38:25 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.648692 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.648699 | controller |       found'
2026-03-14 09:56:26.648705 | controller |     - '    logger.go:42: 09:38:26 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.648711 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.648717 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.648723 | controller |     - '        '
2026-03-14 09:56:26.648729 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.648735 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.648741 | controller |     - '        '
2026-03-14 09:56:26.648751 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.648757 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.648763 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.648770 | controller |     - '          exit 1'
2026-03-14 09:56:26.648776 | controller |     - '        fi'
2026-03-14 09:56:26.648782 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.648788 | controller |     - '        '
2026-03-14 09:56:26.648794 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.648800 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.648806 | controller |       | base64 -d)'
2026-03-14 09:56:26.648823 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.648831 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.648838 | controller |     - '          exit 1'
2026-03-14 09:56:26.648844 | controller |     - '        fi'
2026-03-14 09:56:26.648850 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.648856 | controller |     - '        '
2026-03-14 09:56:26.648862 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.648868 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.648874 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.648880 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.648886 | controller |       --timeout=300s'
2026-03-14 09:56:26.648893 | controller |     - '        '
2026-03-14 09:56:26.648899 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.648905 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.648911 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.648917 | controller |     - '        '
2026-03-14 09:56:26.648923 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.648929 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.648935 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.648941 | controller |     - '        '
2026-03-14 09:56:26.648947 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.648953 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.648959 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.648966 | controller |       available"'
2026-03-14 09:56:26.648972 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.648978 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.648984 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.648990 | controller |     - '          exit 1'
2026-03-14 09:56:26.648996 | controller |     - '        fi'
2026-03-14 09:56:26.649002 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.649008 | controller |     - '        '
2026-03-14 09:56:26.649014 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.649020 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.649027 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.649036 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.649043 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.649049 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.649055 | controller |       -- \'
2026-03-14 09:56:26.649061 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.649067 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.649073 | controller |     - '    logger.go:42: 09:38:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.649080 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.649086 | controller |     - '    logger.go:42: 09:38:26 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.649092 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.649098 | controller |       found'
2026-03-14 09:56:26.649104 | controller |     - '    logger.go:42: 09:38:27 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.649110 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.649116 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.649123 | controller |     - '        '
2026-03-14 09:56:26.649129 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.649135 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.649141 | controller |     - '        '
2026-03-14 09:56:26.649147 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.649153 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.649159 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.649165 | controller |     - '          exit 1'
2026-03-14 09:56:26.649170 | controller |     - '        fi'
2026-03-14 09:56:26.649176 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.649182 | controller |     - '        '
2026-03-14 09:56:26.649188 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.649198 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.649205 | controller |       | base64 -d)'
2026-03-14 09:56:26.649211 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.649228 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.649236 | controller |     - '          exit 1'
2026-03-14 09:56:26.649246 | controller |     - '        fi'
2026-03-14 09:56:26.649252 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.649258 | controller |     - '        '
2026-03-14 09:56:26.649265 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.649270 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.649277 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.649283 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.649289 | controller |       --timeout=300s'
2026-03-14 09:56:26.649295 | controller |     - '        '
2026-03-14 09:56:26.649301 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.649307 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.649313 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.649319 | controller |     - '        '
2026-03-14 09:56:26.649326 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.649335 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.649342 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.649348 | controller |     - '        '
2026-03-14 09:56:26.649354 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.649360 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.649366 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.649372 | controller |       available"'
2026-03-14 09:56:26.649378 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.649384 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.649390 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.649397 | controller |     - '          exit 1'
2026-03-14 09:56:26.649403 | controller |     - '        fi'
2026-03-14 09:56:26.649409 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.649415 | controller |     - '        '
2026-03-14 09:56:26.649421 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.649427 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.649433 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.649439 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.649446 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.649451 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.649458 | controller |       -- \'
2026-03-14 09:56:26.649464 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.649470 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.649480 | controller |     - '    logger.go:42: 09:38:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.649487 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.649493 | controller |     - '    logger.go:42: 09:38:28 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.649498 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.649504 | controller |       found'
2026-03-14 09:56:26.649511 | controller |     - '    logger.go:42: 09:38:29 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.649517 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.649522 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.649529 | controller |     - '        '
2026-03-14 09:56:26.649535 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.649541 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.649547 | controller |     - '        '
2026-03-14 09:56:26.649553 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.649559 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.649565 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.649572 | controller |     - '          exit 1'
2026-03-14 09:56:26.649578 | controller |     - '        fi'
2026-03-14 09:56:26.649583 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.649589 | controller |     - '        '
2026-03-14 09:56:26.649596 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.649626 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.649633 | controller |       | base64 -d)'
2026-03-14 09:56:26.649639 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.649655 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.649663 | controller |     - '          exit 1'
2026-03-14 09:56:26.649669 | controller |     - '        fi'
2026-03-14 09:56:26.649675 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.649681 | controller |     - '        '
2026-03-14 09:56:26.649688 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.649693 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.649699 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.649705 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.649711 | controller |       --timeout=300s'
2026-03-14 09:56:26.649718 | controller |     - '        '
2026-03-14 09:56:26.649724 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.649730 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.649736 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.649742 | controller |     - '        '
2026-03-14 09:56:26.649748 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.649754 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.649761 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.649767 | controller |     - '        '
2026-03-14 09:56:26.649773 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.649779 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.649785 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.649792 | controller |       available"'
2026-03-14 09:56:26.649798 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.649804 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.649810 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.649816 | controller |     - '          exit 1'
2026-03-14 09:56:26.649822 | controller |     - '        fi'
2026-03-14 09:56:26.649829 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.649835 | controller |     - '        '
2026-03-14 09:56:26.649841 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.649847 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.649853 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.649859 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.649865 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.649871 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.649878 | controller |       -- \'
2026-03-14 09:56:26.649884 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.649895 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.649905 | controller |     - '    logger.go:42: 09:38:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.649911 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.649918 | controller |     - '    logger.go:42: 09:38:29 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.649924 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.649930 | controller |       found'
2026-03-14 09:56:26.649937 | controller |     - '    logger.go:42: 09:38:30 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.649943 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.649949 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.649955 | controller |     - '        '
2026-03-14 09:56:26.649961 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.649967 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.649974 | controller |     - '        '
2026-03-14 09:56:26.649980 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.649986 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.649992 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.649998 | controller |     - '          exit 1'
2026-03-14 09:56:26.650005 | controller |     - '        fi'
2026-03-14 09:56:26.650011 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.650021 | controller |     - '        '
2026-03-14 09:56:26.650028 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.650034 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.650042 | controller |       | base64 -d)'
2026-03-14 09:56:26.650048 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.650054 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.650060 | controller |     - '          exit 1'
2026-03-14 09:56:26.650065 | controller |     - '        fi'
2026-03-14 09:56:26.650083 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.650090 | controller |     - '        '
2026-03-14 09:56:26.650096 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.650102 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.650109 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.650291 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.650297 | controller |       --timeout=300s'
2026-03-14 09:56:26.650303 | controller |     - '        '
2026-03-14 09:56:26.650309 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.650315 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.650321 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.650327 | controller |     - '        '
2026-03-14 09:56:26.650333 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.650339 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.650344 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.650350 | controller |     - '        '
2026-03-14 09:56:26.650356 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.650362 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.650373 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.650380 | controller |       available"'
2026-03-14 09:56:26.650386 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.650392 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.650398 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.650404 | controller |     - '          exit 1'
2026-03-14 09:56:26.650410 | controller |     - '        fi'
2026-03-14 09:56:26.650416 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.650422 | controller |     - '        '
2026-03-14 09:56:26.650428 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.650434 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.650440 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.650446 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.650452 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.650458 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.650464 | controller |       -- \'
2026-03-14 09:56:26.650470 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.650476 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.650482 | controller |     - '    logger.go:42: 09:38:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.650489 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.650495 | controller |     - '    logger.go:42: 09:38:30 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.650505 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.650511 | controller |       found'
2026-03-14 09:56:26.650517 | controller |     - '    logger.go:42: 09:38:31 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.650524 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.650534 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.650542 | controller |     - '        '
2026-03-14 09:56:26.650549 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.650556 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.650563 | controller |     - '        '
2026-03-14 09:56:26.650569 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.650575 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.650582 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.650588 | controller |     - '          exit 1'
2026-03-14 09:56:26.650594 | controller |     - '        fi'
2026-03-14 09:56:26.650621 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.650628 | controller |     - '        '
2026-03-14 09:56:26.650634 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.650640 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.650646 | controller |       | base64 -d)'
2026-03-14 09:56:26.650652 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.650658 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.650664 | controller |     - '          exit 1'
2026-03-14 09:56:26.650670 | controller |     - '        fi'
2026-03-14 09:56:26.650687 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.650698 | controller |     - '        '
2026-03-14 09:56:26.650705 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.650711 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.650717 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.650723 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.650729 | controller |       --timeout=300s'
2026-03-14 09:56:26.650735 | controller |     - '        '
2026-03-14 09:56:26.650741 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.650747 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.650753 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.650759 | controller |     - '        '
2026-03-14 09:56:26.650766 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.650772 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.650778 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.650784 | controller |     - '        '
2026-03-14 09:56:26.650790 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.650797 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.650803 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.650809 | controller |       available"'
2026-03-14 09:56:26.650815 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.650821 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.650827 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.650834 | controller |     - '          exit 1'
2026-03-14 09:56:26.650840 | controller |     - '        fi'
2026-03-14 09:56:26.650846 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.650852 | controller |     - '        '
2026-03-14 09:56:26.650858 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.650864 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.650870 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.650875 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.650882 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.650887 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.650894 | controller |       -- \'
2026-03-14 09:56:26.650900 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.650906 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.650912 | controller |     - '    logger.go:42: 09:38:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.650918 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.650925 | controller |     - '    logger.go:42: 09:38:31 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.650931 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.650937 | controller |       found'
2026-03-14 09:56:26.650943 | controller |     - '    logger.go:42: 09:38:32 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.650953 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.650974 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.650980 | controller |     - '        '
2026-03-14 09:56:26.650986 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.650991 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.650997 | controller |     - '        '
2026-03-14 09:56:26.651006 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.651011 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.651016 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.651022 | controller |     - '          exit 1'
2026-03-14 09:56:26.651027 | controller |     - '        fi'
2026-03-14 09:56:26.651034 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.651040 | controller |     - '        '
2026-03-14 09:56:26.651046 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.651052 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.651058 | controller |       | base64 -d)'
2026-03-14 09:56:26.651064 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.651070 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.651076 | controller |     - '          exit 1'
2026-03-14 09:56:26.651082 | controller |     - '        fi'
2026-03-14 09:56:26.651089 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.651095 | controller |     - '        '
2026-03-14 09:56:26.651112 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.651119 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.651125 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.651131 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.651137 | controller |       --timeout=300s'
2026-03-14 09:56:26.651144 | controller |     - '        '
2026-03-14 09:56:26.651150 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.651156 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.651162 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.651168 | controller |     - '        '
2026-03-14 09:56:26.651174 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.651180 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.651187 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.651192 | controller |     - '        '
2026-03-14 09:56:26.651199 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.651205 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.651211 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.651217 | controller |       available"'
2026-03-14 09:56:26.651223 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.651229 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.651239 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.651245 | controller |     - '          exit 1'
2026-03-14 09:56:26.651251 | controller |     - '        fi'
2026-03-14 09:56:26.651260 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.651267 | controller |     - '        '
2026-03-14 09:56:26.651273 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.651279 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.651285 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.651291 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.651297 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.651303 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.651309 | controller |       -- \'
2026-03-14 09:56:26.651316 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.651322 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.651329 | controller |     - '    logger.go:42: 09:38:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.651335 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.651341 | controller |     - '    logger.go:42: 09:38:32 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.651347 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.651357 | controller |       found'
2026-03-14 09:56:26.651364 | controller |     - '    logger.go:42: 09:38:33 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.651370 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.651376 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.651381 | controller |     - '        '
2026-03-14 09:56:26.651386 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.651392 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.651398 | controller |     - '        '
2026-03-14 09:56:26.651404 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.651411 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.651417 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.651423 | controller |     - '          exit 1'
2026-03-14 09:56:26.651429 | controller |     - '        fi'
2026-03-14 09:56:26.651435 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.651441 | controller |     - '        '
2026-03-14 09:56:26.651447 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.651453 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.651459 | controller |       | base64 -d)'
2026-03-14 09:56:26.651466 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.651471 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.651478 | controller |     - '          exit 1'
2026-03-14 09:56:26.651484 | controller |     - '        fi'
2026-03-14 09:56:26.651490 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.651496 | controller |     - '        '
2026-03-14 09:56:26.651502 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.651519 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.651527 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.651533 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.651539 | controller |       --timeout=300s'
2026-03-14 09:56:26.651545 | controller |     - '        '
2026-03-14 09:56:26.651555 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.651561 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.651567 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.651573 | controller |     - '        '
2026-03-14 09:56:26.651580 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.651586 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.651592 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.651620 | controller |     - '        '
2026-03-14 09:56:26.651628 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.651634 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.651640 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.651646 | controller |       available"'
2026-03-14 09:56:26.651652 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.651658 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.651664 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.651670 | controller |     - '          exit 1'
2026-03-14 09:56:26.651676 | controller |     - '        fi'
2026-03-14 09:56:26.651682 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.651688 | controller |     - '        '
2026-03-14 09:56:26.651694 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.651700 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.651706 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.651712 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.651718 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.651724 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.651730 | controller |       -- \'
2026-03-14 09:56:26.651736 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.651743 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.651749 | controller |     - '    logger.go:42: 09:38:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.651755 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.651761 | controller |     - '    logger.go:42: 09:38:34 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.651767 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.651773 | controller |       found'
2026-03-14 09:56:26.651779 | controller |     - '    logger.go:42: 09:38:35 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.651786 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.651792 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.651798 | controller |     - '        '
2026-03-14 09:56:26.651804 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.651810 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.651816 | controller |     - '        '
2026-03-14 09:56:26.651823 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.651833 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.651839 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.651845 | controller |     - '          exit 1'
2026-03-14 09:56:26.651851 | controller |     - '        fi'
2026-03-14 09:56:26.651857 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.651863 | controller |     - '        '
2026-03-14 09:56:26.651869 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.651875 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.651881 | controller |       | base64 -d)'
2026-03-14 09:56:26.651887 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.651893 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.651900 | controller |     - '          exit 1'
2026-03-14 09:56:26.651906 | controller |     - '        fi'
2026-03-14 09:56:26.651912 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.651918 | controller |     - '        '
2026-03-14 09:56:26.651924 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.651941 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.651948 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.651958 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.651965 | controller |       --timeout=300s'
2026-03-14 09:56:26.651971 | controller |     - '        '
2026-03-14 09:56:26.651977 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.651983 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.651989 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.651995 | controller |     - '        '
2026-03-14 09:56:26.652001 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.652007 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.652013 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.652019 | controller |     - '        '
2026-03-14 09:56:26.652025 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.652031 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.652037 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.652043 | controller |       available"'
2026-03-14 09:56:26.652049 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.652056 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.652062 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.652068 | controller |     - '          exit 1'
2026-03-14 09:56:26.652074 | controller |     - '        fi'
2026-03-14 09:56:26.652080 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.652087 | controller |     - '        '
2026-03-14 09:56:26.652093 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.652099 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.652105 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.652111 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.652293 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.652299 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.652309 | controller |       -- \'
2026-03-14 09:56:26.652315 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.652322 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.652328 | controller |     - '    logger.go:42: 09:38:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.652334 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.652341 | controller |     - '    logger.go:42: 09:38:35 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.652346 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.652353 | controller |       found'
2026-03-14 09:56:26.652359 | controller |     - '    logger.go:42: 09:38:36 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.652371 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.652377 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.652383 | controller |     - '        '
2026-03-14 09:56:26.652390 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.652396 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.652402 | controller |     - '        '
2026-03-14 09:56:26.652409 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.652415 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.652422 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.652428 | controller |     - '          exit 1'
2026-03-14 09:56:26.652434 | controller |     - '        fi'
2026-03-14 09:56:26.652440 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.652447 | controller |     - '        '
2026-03-14 09:56:26.652453 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.652459 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.652466 | controller |       | base64 -d)'
2026-03-14 09:56:26.652472 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.652492 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.652499 | controller |     - '          exit 1'
2026-03-14 09:56:26.652504 | controller |     - '        fi'
2026-03-14 09:56:26.652517 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.652524 | controller |     - '        '
2026-03-14 09:56:26.652530 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.652536 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.652554 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.652561 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.652568 | controller |       --timeout=300s'
2026-03-14 09:56:26.652574 | controller |     - '        '
2026-03-14 09:56:26.652580 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.652587 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.652593 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.652617 | controller |     - '        '
2026-03-14 09:56:26.652625 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.652635 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.652641 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.652648 | controller |     - '        '
2026-03-14 09:56:26.652654 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.652660 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.652666 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.652672 | controller |       available"'
2026-03-14 09:56:26.652682 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.652688 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.652694 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.652700 | controller |     - '          exit 1'
2026-03-14 09:56:26.652706 | controller |     - '        fi'
2026-03-14 09:56:26.652712 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.652718 | controller |     - '        '
2026-03-14 09:56:26.652724 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.652730 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.652737 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.652743 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.652749 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.652755 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.652761 | controller |       -- \'
2026-03-14 09:56:26.652767 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.652773 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.652780 | controller |     - '    logger.go:42: 09:38:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.652785 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.652791 | controller |     - '    logger.go:42: 09:38:36 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.652798 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.652804 | controller |       found'
2026-03-14 09:56:26.652810 | controller |     - '    logger.go:42: 09:38:37 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.652816 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.652822 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.652828 | controller |     - '        '
2026-03-14 09:56:26.652834 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.652840 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.652847 | controller |     - '        '
2026-03-14 09:56:26.652853 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.652858 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.652865 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.652871 | controller |     - '          exit 1'
2026-03-14 09:56:26.652877 | controller |     - '        fi'
2026-03-14 09:56:26.652883 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.652889 | controller |     - '        '
2026-03-14 09:56:26.652895 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.652902 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.652911 | controller |       | base64 -d)'
2026-03-14 09:56:26.652918 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.652924 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.652930 | controller |     - '          exit 1'
2026-03-14 09:56:26.652936 | controller |     - '        fi'
2026-03-14 09:56:26.652942 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.652948 | controller |     - '        '
2026-03-14 09:56:26.652954 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.652960 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.652967 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.652984 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.652992 | controller |       --timeout=300s'
2026-03-14 09:56:26.652998 | controller |     - '        '
2026-03-14 09:56:26.653004 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.653010 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.653017 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.653023 | controller |     - '        '
2026-03-14 09:56:26.653029 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.653036 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.653041 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.653048 | controller |     - '        '
2026-03-14 09:56:26.653054 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.653060 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.653066 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.653072 | controller |       available"'
2026-03-14 09:56:26.653078 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.653084 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.653090 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.653096 | controller |     - '          exit 1'
2026-03-14 09:56:26.653102 | controller |     - '        fi'
2026-03-14 09:56:26.653108 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.653115 | controller |     - '        '
2026-03-14 09:56:26.653121 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.653126 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.653133 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.653139 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.653153 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.653159 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.653166 | controller |       -- \'
2026-03-14 09:56:26.653171 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.653184 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.653194 | controller |     - '    logger.go:42: 09:38:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.653200 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.653207 | controller |     - '    logger.go:42: 09:38:37 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.653213 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.653219 | controller |       found'
2026-03-14 09:56:26.653225 | controller |     - '    logger.go:42: 09:38:38 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.653231 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.653237 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.653244 | controller |     - '        '
2026-03-14 09:56:26.653250 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.653261 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.653267 | controller |     - '        '
2026-03-14 09:56:26.653274 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.653280 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.653286 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.653292 | controller |     - '          exit 1'
2026-03-14 09:56:26.653298 | controller |     - '        fi'
2026-03-14 09:56:26.653304 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.653310 | controller |     - '        '
2026-03-14 09:56:26.653317 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.653327 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.653334 | controller |       | base64 -d)'
2026-03-14 09:56:26.653341 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.653348 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.653355 | controller |     - '          exit 1'
2026-03-14 09:56:26.653363 | controller |     - '        fi'
2026-03-14 09:56:26.653369 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.653375 | controller |     - '        '
2026-03-14 09:56:26.653382 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.653388 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.653395 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.653412 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.653420 | controller |       --timeout=300s'
2026-03-14 09:56:26.653427 | controller |     - '        '
2026-03-14 09:56:26.653433 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.653439 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.653449 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.653455 | controller |     - '        '
2026-03-14 09:56:26.653460 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.653466 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.653471 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.653477 | controller |     - '        '
2026-03-14 09:56:26.653483 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.653489 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.653496 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.653505 | controller |       available"'
2026-03-14 09:56:26.653511 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.653518 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.653524 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.653530 | controller |     - '          exit 1'
2026-03-14 09:56:26.653536 | controller |     - '        fi'
2026-03-14 09:56:26.653542 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.653549 | controller |     - '        '
2026-03-14 09:56:26.653555 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.653561 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.653567 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.653573 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.653579 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.653586 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.653592 | controller |       -- \'
2026-03-14 09:56:26.653613 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.653624 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.653631 | controller |     - '    logger.go:42: 09:38:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.653644 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.653651 | controller |     - '    logger.go:42: 09:38:38 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.653658 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.653664 | controller |       found'
2026-03-14 09:56:26.653671 | controller |     - '    logger.go:42: 09:38:39 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.653677 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.653683 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.653689 | controller |     - '        '
2026-03-14 09:56:26.653695 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.653701 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.653707 | controller |     - '        '
2026-03-14 09:56:26.653714 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.653720 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.653726 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.653732 | controller |     - '          exit 1'
2026-03-14 09:56:26.653739 | controller |     - '        fi'
2026-03-14 09:56:26.653745 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.653763 | controller |     - '        '
2026-03-14 09:56:26.653771 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.653777 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.653784 | controller |       | base64 -d)'
2026-03-14 09:56:26.653797 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.653832 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.653839 | controller |     - '          exit 1'
2026-03-14 09:56:26.653847 | controller |     - '        fi'
2026-03-14 09:56:26.653853 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.653859 | controller |     - '        '
2026-03-14 09:56:26.653876 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.653887 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.653897 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.653913 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.653927 | controller |       --timeout=300s'
2026-03-14 09:56:26.653934 | controller |     - '        '
2026-03-14 09:56:26.653940 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.653947 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.653958 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.653965 | controller |     - '        '
2026-03-14 09:56:26.653972 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.653978 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.653989 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.653997 | controller |     - '        '
2026-03-14 09:56:26.654003 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.654009 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.654016 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.654023 | controller |       available"'
2026-03-14 09:56:26.654029 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.654035 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.654041 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.654047 | controller |     - '          exit 1'
2026-03-14 09:56:26.654053 | controller |     - '        fi'
2026-03-14 09:56:26.654059 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.654065 | controller |     - '        '
2026-03-14 09:56:26.654072 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.654078 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.654084 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.654090 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.654096 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.654102 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.654108 | controller |       -- \'
2026-03-14 09:56:26.654115 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.654120 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.654126 | controller |     - '    logger.go:42: 09:38:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.654132 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.654138 | controller |     - '    logger.go:42: 09:38:39 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.654144 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.654150 | controller |       found'
2026-03-14 09:56:26.654156 | controller |     - '    logger.go:42: 09:38:40 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.654166 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.654173 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.654179 | controller |     - '        '
2026-03-14 09:56:26.654185 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.654192 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.654202 | controller |     - '        '
2026-03-14 09:56:26.654208 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.654214 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.654220 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.654227 | controller |     - '          exit 1'
2026-03-14 09:56:26.654233 | controller |     - '        fi'
2026-03-14 09:56:26.654239 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.654245 | controller |     - '        '
2026-03-14 09:56:26.654251 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.654257 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.654263 | controller |       | base64 -d)'
2026-03-14 09:56:26.654270 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.654276 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.654282 | controller |     - '          exit 1'
2026-03-14 09:56:26.654288 | controller |     - '        fi'
2026-03-14 09:56:26.654294 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.654304 | controller |     - '        '
2026-03-14 09:56:26.654310 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.654316 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.654323 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.654329 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.654335 | controller |       --timeout=300s'
2026-03-14 09:56:26.654341 | controller |     - '        '
2026-03-14 09:56:26.654358 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.654369 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.654375 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.654382 | controller |     - '        '
2026-03-14 09:56:26.654388 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.654394 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.654400 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.654406 | controller |     - '        '
2026-03-14 09:56:26.654412 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.654418 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.654424 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.654430 | controller |       available"'
2026-03-14 09:56:26.654436 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.654442 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.654449 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.654455 | controller |     - '          exit 1'
2026-03-14 09:56:26.654461 | controller |     - '        fi'
2026-03-14 09:56:26.654467 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.654477 | controller |     - '        '
2026-03-14 09:56:26.654483 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.654489 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.654495 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.654501 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.654507 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.654513 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.654519 | controller |       -- \'
2026-03-14 09:56:26.654525 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.654531 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.654538 | controller |     - '    logger.go:42: 09:38:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.654543 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.654549 | controller |     - '    logger.go:42: 09:38:40 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.654555 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.654569 | controller |       found'
2026-03-14 09:56:26.654576 | controller |     - '    logger.go:42: 09:38:41 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.654587 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.654594 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.654622 | controller |     - '        '
2026-03-14 09:56:26.654630 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.654635 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.654641 | controller |     - '        '
2026-03-14 09:56:26.654647 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.654653 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.654658 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.654664 | controller |     - '          exit 1'
2026-03-14 09:56:26.654671 | controller |     - '        fi'
2026-03-14 09:56:26.654677 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.654683 | controller |     - '        '
2026-03-14 09:56:26.654689 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.654695 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.654700 | controller |       | base64 -d)'
2026-03-14 09:56:26.654706 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.654713 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.654719 | controller |     - '          exit 1'
2026-03-14 09:56:26.654725 | controller |     - '        fi'
2026-03-14 09:56:26.654731 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.654737 | controller |     - '        '
2026-03-14 09:56:26.654743 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.654749 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.654755 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.654761 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.654768 | controller |       --timeout=300s'
2026-03-14 09:56:26.654774 | controller |     - '        '
2026-03-14 09:56:26.654791 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.654805 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.654812 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.654818 | controller |     - '        '
2026-03-14 09:56:26.654824 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.654831 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.654837 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.654843 | controller |     - '        '
2026-03-14 09:56:26.654849 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.654856 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.654862 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.654869 | controller |       available"'
2026-03-14 09:56:26.654875 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.654881 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.654887 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.654894 | controller |     - '          exit 1'
2026-03-14 09:56:26.654900 | controller |     - '        fi'
2026-03-14 09:56:26.654906 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.654912 | controller |     - '        '
2026-03-14 09:56:26.654919 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.654925 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.654931 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.654943 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.654949 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.654956 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.654979 | controller |       -- \'
2026-03-14 09:56:26.654986 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.654993 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.654999 | controller |     - '    logger.go:42: 09:38:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.655005 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.655012 | controller |     - '    logger.go:42: 09:38:42 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.655018 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.655023 | controller |       found'
2026-03-14 09:56:26.655029 | controller |     - '    logger.go:42: 09:38:43 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.655035 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.655040 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.655046 | controller |     - '        '
2026-03-14 09:56:26.655051 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.655057 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.655062 | controller |     - '        '
2026-03-14 09:56:26.655067 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.655076 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.655084 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.655090 | controller |     - '          exit 1'
2026-03-14 09:56:26.655096 | controller |     - '        fi'
2026-03-14 09:56:26.655101 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.655107 | controller |     - '        '
2026-03-14 09:56:26.655112 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.655118 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.655122 | controller |       | base64 -d)'
2026-03-14 09:56:26.655126 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.655131 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.655135 | controller |     - '          exit 1'
2026-03-14 09:56:26.655139 | controller |     - '        fi'
2026-03-14 09:56:26.655144 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.655148 | controller |     - '        '
2026-03-14 09:56:26.655232 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.655239 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.655244 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.655248 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.655252 | controller |       --timeout=300s'
2026-03-14 09:56:26.655256 | controller |     - '        '
2026-03-14 09:56:26.655261 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.655275 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.655286 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.655291 | controller |     - '        '
2026-03-14 09:56:26.655295 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.655299 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.655304 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.655308 | controller |     - '        '
2026-03-14 09:56:26.655312 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.655316 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.655321 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.655325 | controller |       available"'
2026-03-14 09:56:26.655329 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.655334 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.655341 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.655345 | controller |     - '          exit 1'
2026-03-14 09:56:26.655349 | controller |     - '        fi'
2026-03-14 09:56:26.655354 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.655358 | controller |     - '        '
2026-03-14 09:56:26.655362 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.655367 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.655371 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.655375 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.655379 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.655386 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.655390 | controller |       -- \'
2026-03-14 09:56:26.655394 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.655399 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.655403 | controller |     - '    logger.go:42: 09:38:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.655407 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.655412 | controller |     - '    logger.go:42: 09:38:43 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.655416 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.655420 | controller |       found'
2026-03-14 09:56:26.655425 | controller |     - '    logger.go:42: 09:38:44 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.655429 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.655433 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.655438 | controller |     - '        '
2026-03-14 09:56:26.655442 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.655446 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.655450 | controller |     - '        '
2026-03-14 09:56:26.655455 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.655459 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.655463 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.655468 | controller |     - '          exit 1'
2026-03-14 09:56:26.655472 | controller |     - '        fi'
2026-03-14 09:56:26.655476 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.655481 | controller |     - '        '
2026-03-14 09:56:26.655485 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.655489 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.655494 | controller |       | base64 -d)'
2026-03-14 09:56:26.655498 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.655502 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.655506 | controller |     - '          exit 1'
2026-03-14 09:56:26.655511 | controller |     - '        fi'
2026-03-14 09:56:26.655515 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.655520 | controller |     - '        '
2026-03-14 09:56:26.655524 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.655528 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.655532 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.655537 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.655541 | controller |       --timeout=300s'
2026-03-14 09:56:26.655545 | controller |     - '        '
2026-03-14 09:56:26.655550 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.655561 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.655565 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.655570 | controller |     - '        '
2026-03-14 09:56:26.655574 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.655579 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.655585 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.655590 | controller |     - '        '
2026-03-14 09:56:26.655594 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.655612 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.655628 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.655633 | controller |       available"'
2026-03-14 09:56:26.655637 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.655644 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.655649 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.655653 | controller |     - '          exit 1'
2026-03-14 09:56:26.655657 | controller |     - '        fi'
2026-03-14 09:56:26.655662 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.655666 | controller |     - '        '
2026-03-14 09:56:26.655670 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.655675 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.655679 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.655683 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.655688 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.655692 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.655696 | controller |       -- \'
2026-03-14 09:56:26.655701 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.655705 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.655709 | controller |     - '    logger.go:42: 09:38:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.655713 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.655718 | controller |     - '    logger.go:42: 09:38:44 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.655722 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.655726 | controller |       found'
2026-03-14 09:56:26.655731 | controller |     - '    logger.go:42: 09:38:45 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.655735 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.655739 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.655744 | controller |     - '        '
2026-03-14 09:56:26.655748 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.655752 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.655757 | controller |     - '        '
2026-03-14 09:56:26.655761 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.655765 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.655772 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.655777 | controller |     - '          exit 1'
2026-03-14 09:56:26.655781 | controller |     - '        fi'
2026-03-14 09:56:26.655787 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.655791 | controller |     - '        '
2026-03-14 09:56:26.655796 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.655800 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.655806 | controller |       | base64 -d)'
2026-03-14 09:56:26.655811 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.655815 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.655819 | controller |     - '          exit 1'
2026-03-14 09:56:26.655824 | controller |     - '        fi'
2026-03-14 09:56:26.655828 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.655832 | controller |     - '        '
2026-03-14 09:56:26.655837 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.655841 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.655847 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.655851 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.655856 | controller |       --timeout=300s'
2026-03-14 09:56:26.655860 | controller |     - '        '
2026-03-14 09:56:26.655864 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.655869 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.655881 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.655886 | controller |     - '        '
2026-03-14 09:56:26.655891 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.655895 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.655900 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.655904 | controller |     - '        '
2026-03-14 09:56:26.655908 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.655913 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.655917 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.655921 | controller |       available"'
2026-03-14 09:56:26.655926 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.655930 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.655934 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.655939 | controller |     - '          exit 1'
2026-03-14 09:56:26.655943 | controller |     - '        fi'
2026-03-14 09:56:26.655947 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.655952 | controller |     - '        '
2026-03-14 09:56:26.655956 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.655960 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.655965 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.655969 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.655973 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.655978 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.655982 | controller |       -- \'
2026-03-14 09:56:26.655987 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.655991 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.655995 | controller |     - '    logger.go:42: 09:38:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.656001 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.656006 | controller |     - '    logger.go:42: 09:38:45 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.656010 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.656015 | controller |       found'
2026-03-14 09:56:26.656019 | controller |     - '    logger.go:42: 09:38:46 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.656029 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.656034 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.656038 | controller |     - '        '
2026-03-14 09:56:26.656043 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.656047 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.656051 | controller |     - '        '
2026-03-14 09:56:26.656056 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.656060 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.656065 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.656069 | controller |     - '          exit 1'
2026-03-14 09:56:26.656073 | controller |     - '        fi'
2026-03-14 09:56:26.656078 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.656084 | controller |     - '        '
2026-03-14 09:56:26.656088 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.656093 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.656099 | controller |       | base64 -d)'
2026-03-14 09:56:26.656103 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.656107 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.656112 | controller |     - '          exit 1'
2026-03-14 09:56:26.656116 | controller |     - '        fi'
2026-03-14 09:56:26.656120 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.656125 | controller |     - '        '
2026-03-14 09:56:26.656129 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.656133 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.656138 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.656142 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.656146 | controller |       --timeout=300s'
2026-03-14 09:56:26.656150 | controller |     - '        '
2026-03-14 09:56:26.656155 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.656159 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.656170 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.656179 | controller |     - '        '
2026-03-14 09:56:26.656183 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.656187 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.656192 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.656196 | controller |     - '        '
2026-03-14 09:56:26.656201 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.656205 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.656209 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.656215 | controller |       available"'
2026-03-14 09:56:26.656220 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.656224 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.656228 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.656233 | controller |     - '          exit 1'
2026-03-14 09:56:26.656237 | controller |     - '        fi'
2026-03-14 09:56:26.656241 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.656246 | controller |     - '        '
2026-03-14 09:56:26.656250 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.656254 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.656259 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.656263 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.656267 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.656272 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.656276 | controller |       -- \'
2026-03-14 09:56:26.656280 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.656285 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.656289 | controller |     - '    logger.go:42: 09:38:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.656293 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.656298 | controller |     - '    logger.go:42: 09:38:46 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.656302 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.656306 | controller |       found'
2026-03-14 09:56:26.656310 | controller |     - '    logger.go:42: 09:38:47 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.656315 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.656319 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.656323 | controller |     - '        '
2026-03-14 09:56:26.656328 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.656332 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.656337 | controller |     - '        '
2026-03-14 09:56:26.656343 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.656347 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.656352 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.656358 | controller |     - '          exit 1'
2026-03-14 09:56:26.656363 | controller |     - '        fi'
2026-03-14 09:56:26.656367 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.656372 | controller |     - '        '
2026-03-14 09:56:26.656376 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.656380 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.656384 | controller |       | base64 -d)'
2026-03-14 09:56:26.656389 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.656393 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.656397 | controller |     - '          exit 1'
2026-03-14 09:56:26.656402 | controller |     - '        fi'
2026-03-14 09:56:26.656406 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.656410 | controller |     - '        '
2026-03-14 09:56:26.656415 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.656421 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.656425 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.656431 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.656436 | controller |       --timeout=300s'
2026-03-14 09:56:26.656440 | controller |     - '        '
2026-03-14 09:56:26.656444 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.656449 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.656453 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.656464 | controller |     - '        '
2026-03-14 09:56:26.656468 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.656473 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.656477 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.656481 | controller |     - '        '
2026-03-14 09:56:26.656486 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.656490 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.656494 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.656499 | controller |       available"'
2026-03-14 09:56:26.656503 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.656507 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.656511 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.656516 | controller |     - '          exit 1'
2026-03-14 09:56:26.656520 | controller |     - '        fi'
2026-03-14 09:56:26.656524 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.656529 | controller |     - '        '
2026-03-14 09:56:26.656533 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.656537 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.656542 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.656546 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.656550 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.656554 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.656559 | controller |       -- \'
2026-03-14 09:56:26.656563 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.656567 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.656572 | controller |     - '    logger.go:42: 09:38:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.656576 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.656580 | controller |     - '    logger.go:42: 09:38:48 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.656585 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.656589 | controller |       found'
2026-03-14 09:56:26.656593 | controller |     - '    logger.go:42: 09:38:49 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.656612 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.656619 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.656623 | controller |     - '        '
2026-03-14 09:56:26.656628 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.656632 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.656636 | controller |     - '        '
2026-03-14 09:56:26.656641 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.656645 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.656649 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.656654 | controller |     - '          exit 1'
2026-03-14 09:56:26.656658 | controller |     - '        fi'
2026-03-14 09:56:26.656662 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.656667 | controller |     - '        '
2026-03-14 09:56:26.656671 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.656677 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.656681 | controller |       | base64 -d)'
2026-03-14 09:56:26.656686 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.656690 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.656694 | controller |     - '          exit 1'
2026-03-14 09:56:26.656699 | controller |     - '        fi'
2026-03-14 09:56:26.656703 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.656707 | controller |     - '        '
2026-03-14 09:56:26.656712 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.656716 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.656720 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.656725 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.656729 | controller |       --timeout=300s'
2026-03-14 09:56:26.656733 | controller |     - '        '
2026-03-14 09:56:26.656738 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.656742 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.656746 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.656751 | controller |     - '        '
2026-03-14 09:56:26.656762 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.656767 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.656771 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.656775 | controller |     - '        '
2026-03-14 09:56:26.656780 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.656784 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.656789 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.656793 | controller |       available"'
2026-03-14 09:56:26.656797 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.656801 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.656806 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.656810 | controller |     - '          exit 1'
2026-03-14 09:56:26.656815 | controller |     - '        fi'
2026-03-14 09:56:26.656821 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.656825 | controller |     - '        '
2026-03-14 09:56:26.656831 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.656836 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.656840 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.656844 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.656849 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.656853 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.656857 | controller |       -- \'
2026-03-14 09:56:26.656862 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.656866 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.656870 | controller |     - '    logger.go:42: 09:38:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.656875 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.656879 | controller |     - '    logger.go:42: 09:38:49 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.656883 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.656888 | controller |       found'
2026-03-14 09:56:26.656892 | controller |     - '    logger.go:42: 09:38:50 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.656896 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.656901 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.656905 | controller |     - '        '
2026-03-14 09:56:26.656909 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.656914 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.656918 | controller |     - '        '
2026-03-14 09:56:26.656922 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.656926 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.656931 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.656935 | controller |     - '          exit 1'
2026-03-14 09:56:26.656939 | controller |     - '        fi'
2026-03-14 09:56:26.656944 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.656948 | controller |     - '        '
2026-03-14 09:56:26.656952 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.656957 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.656961 | controller |       | base64 -d)'
2026-03-14 09:56:26.656965 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.656969 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.656974 | controller |     - '          exit 1'
2026-03-14 09:56:26.656978 | controller |     - '        fi'
2026-03-14 09:56:26.656982 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.656987 | controller |     - '        '
2026-03-14 09:56:26.656991 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.656995 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.657000 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.657004 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.657010 | controller |       --timeout=300s'
2026-03-14 09:56:26.657014 | controller |     - '        '
2026-03-14 09:56:26.657019 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.657025 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.657029 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.657033 | controller |     - '        '
2026-03-14 09:56:26.657044 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.657049 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.657053 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.657057 | controller |     - '        '
2026-03-14 09:56:26.657062 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.657066 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.657071 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.657075 | controller |       available"'
2026-03-14 09:56:26.657079 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.657084 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.657088 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.657092 | controller |     - '          exit 1'
2026-03-14 09:56:26.657097 | controller |     - '        fi'
2026-03-14 09:56:26.657101 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.657105 | controller |     - '        '
2026-03-14 09:56:26.657110 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.657114 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.657118 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.657123 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.657127 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.657133 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.657138 | controller |       -- \'
2026-03-14 09:56:26.657142 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.657146 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.657151 | controller |     - '    logger.go:42: 09:38:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.657155 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.657160 | controller |     - '    logger.go:42: 09:38:50 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.657164 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.657168 | controller |       found'
2026-03-14 09:56:26.657173 | controller |     - '    logger.go:42: 09:38:51 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.657177 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.657181 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.657185 | controller |     - '        '
2026-03-14 09:56:26.657190 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.657194 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.657198 | controller |     - '        '
2026-03-14 09:56:26.657203 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.657207 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.657211 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.657218 | controller |     - '          exit 1'
2026-03-14 09:56:26.657222 | controller |     - '        fi'
2026-03-14 09:56:26.657226 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.657231 | controller |     - '        '
2026-03-14 09:56:26.657235 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.657239 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.657244 | controller |       | base64 -d)'
2026-03-14 09:56:26.657248 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.657252 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.657257 | controller |     - '          exit 1'
2026-03-14 09:56:26.657261 | controller |     - '        fi'
2026-03-14 09:56:26.657265 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.657269 | controller |     - '        '
2026-03-14 09:56:26.657274 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.657278 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.657282 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.657287 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.657291 | controller |       --timeout=300s'
2026-03-14 09:56:26.657295 | controller |     - '        '
2026-03-14 09:56:26.657300 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.657304 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.657308 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.657313 | controller |     - '        '
2026-03-14 09:56:26.657317 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.657328 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.657332 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.657337 | controller |     - '        '
2026-03-14 09:56:26.657341 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.657346 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.657350 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.657354 | controller |       available"'
2026-03-14 09:56:26.657359 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.657363 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.657367 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.657372 | controller |     - '          exit 1'
2026-03-14 09:56:26.657376 | controller |     - '        fi'
2026-03-14 09:56:26.657380 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.657385 | controller |     - '        '
2026-03-14 09:56:26.657389 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.657393 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.657398 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.657402 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.657406 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.657413 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.657419 | controller |       -- \'
2026-03-14 09:56:26.657423 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.657428 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.657432 | controller |     - '    logger.go:42: 09:38:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.657436 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.657441 | controller |     - '    logger.go:42: 09:38:51 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.657445 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.657449 | controller |       found'
2026-03-14 09:56:26.657454 | controller |     - '    logger.go:42: 09:38:52 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.657458 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.657462 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.657467 | controller |     - '        '
2026-03-14 09:56:26.657471 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.657475 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.657480 | controller |     - '        '
2026-03-14 09:56:26.657484 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.657488 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.657492 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.657497 | controller |     - '          exit 1'
2026-03-14 09:56:26.657501 | controller |     - '        fi'
2026-03-14 09:56:26.657505 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.657510 | controller |     - '        '
2026-03-14 09:56:26.657514 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.657518 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.657523 | controller |       | base64 -d)'
2026-03-14 09:56:26.657527 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.657531 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.657536 | controller |     - '          exit 1'
2026-03-14 09:56:26.657540 | controller |     - '        fi'
2026-03-14 09:56:26.657544 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.657549 | controller |     - '        '
2026-03-14 09:56:26.657553 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.657557 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.657561 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.657566 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.657570 | controller |       --timeout=300s'
2026-03-14 09:56:26.657574 | controller |     - '        '
2026-03-14 09:56:26.657579 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.657583 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.657587 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.657592 | controller |     - '        '
2026-03-14 09:56:26.657606 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.657629 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.657639 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.657644 | controller |     - '        '
2026-03-14 09:56:26.657648 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.657652 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.657657 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.657661 | controller |       available"'
2026-03-14 09:56:26.657665 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.657670 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.657674 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.657678 | controller |     - '          exit 1'
2026-03-14 09:56:26.657683 | controller |     - '        fi'
2026-03-14 09:56:26.657687 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.657691 | controller |     - '        '
2026-03-14 09:56:26.657700 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.657705 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.657710 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.657714 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.657718 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.657723 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.657727 | controller |       -- \'
2026-03-14 09:56:26.657731 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.657738 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.657743 | controller |     - '    logger.go:42: 09:38:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.657747 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.657751 | controller |     - '    logger.go:42: 09:38:52 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.657756 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.657760 | controller |       found'
2026-03-14 09:56:26.657764 | controller |     - '    logger.go:42: 09:38:53 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.657769 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.657776 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.657782 | controller |     - '        '
2026-03-14 09:56:26.657800 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.657807 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.657813 | controller |     - '        '
2026-03-14 09:56:26.657819 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.657825 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.657831 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.657835 | controller |     - '          exit 1'
2026-03-14 09:56:26.657840 | controller |     - '        fi'
2026-03-14 09:56:26.657844 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.657848 | controller |     - '        '
2026-03-14 09:56:26.657852 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.657857 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.657861 | controller |       | base64 -d)'
2026-03-14 09:56:26.657869 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.657874 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.657878 | controller |     - '          exit 1'
2026-03-14 09:56:26.657882 | controller |     - '        fi'
2026-03-14 09:56:26.657887 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.657891 | controller |     - '        '
2026-03-14 09:56:26.657895 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.657900 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.657904 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.657908 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.657913 | controller |       --timeout=300s'
2026-03-14 09:56:26.657917 | controller |     - '        '
2026-03-14 09:56:26.657924 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.657929 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.657933 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.657937 | controller |     - '        '
2026-03-14 09:56:26.657942 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.657946 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.657960 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.657965 | controller |     - '        '
2026-03-14 09:56:26.657970 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.657974 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.657978 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.657983 | controller |       available"'
2026-03-14 09:56:26.657987 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.657991 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.657997 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.658002 | controller |     - '          exit 1'
2026-03-14 09:56:26.658006 | controller |     - '        fi'
2026-03-14 09:56:26.658010 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.658015 | controller |     - '        '
2026-03-14 09:56:26.658019 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.658023 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.658027 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.658031 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.658036 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.658040 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.658044 | controller |       -- \'
2026-03-14 09:56:26.658048 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.658053 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.658057 | controller |     - '    logger.go:42: 09:38:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.658065 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.658069 | controller |     - '    logger.go:42: 09:38:53 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.658073 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.658078 | controller |       found'
2026-03-14 09:56:26.658082 | controller |     - '    logger.go:42: 09:38:54 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.658086 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.658091 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.658095 | controller |     - '        '
2026-03-14 09:56:26.658099 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.658103 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.658108 | controller |     - '        '
2026-03-14 09:56:26.658112 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.658116 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.658120 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.658124 | controller |     - '          exit 1'
2026-03-14 09:56:26.658129 | controller |     - '        fi'
2026-03-14 09:56:26.658133 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.658137 | controller |     - '        '
2026-03-14 09:56:26.658141 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.658146 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.658150 | controller |       | base64 -d)'
2026-03-14 09:56:26.658154 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.658158 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.658163 | controller |     - '          exit 1'
2026-03-14 09:56:26.658167 | controller |     - '        fi'
2026-03-14 09:56:26.658171 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.658175 | controller |     - '        '
2026-03-14 09:56:26.658180 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.658184 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.658188 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.658192 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.658196 | controller |       --timeout=300s'
2026-03-14 09:56:26.658201 | controller |     - '        '
2026-03-14 09:56:26.658205 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.658209 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.658214 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.658218 | controller |     - '        '
2026-03-14 09:56:26.658222 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.658226 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.658240 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.658245 | controller |     - '        '
2026-03-14 09:56:26.658249 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.658254 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.658258 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.658262 | controller |       available"'
2026-03-14 09:56:26.658266 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.658273 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.658277 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.658281 | controller |     - '          exit 1'
2026-03-14 09:56:26.658285 | controller |     - '        fi'
2026-03-14 09:56:26.658290 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.658294 | controller |     - '        '
2026-03-14 09:56:26.658298 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.658302 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.658307 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.658311 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.658315 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.658319 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.658323 | controller |       -- \'
2026-03-14 09:56:26.658328 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.658332 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.658336 | controller |     - '    logger.go:42: 09:38:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.658340 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.658345 | controller |     - '    logger.go:42: 09:38:55 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.658349 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.658353 | controller |       found'
2026-03-14 09:56:26.658357 | controller |     - '    logger.go:42: 09:38:56 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.658362 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.658366 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.658370 | controller |     - '        '
2026-03-14 09:56:26.658374 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.658379 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.658383 | controller |     - '        '
2026-03-14 09:56:26.658387 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.658391 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.658396 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.658400 | controller |     - '          exit 1'
2026-03-14 09:56:26.658404 | controller |     - '        fi'
2026-03-14 09:56:26.658408 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.658413 | controller |     - '        '
2026-03-14 09:56:26.658417 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.658421 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.658425 | controller |       | base64 -d)'
2026-03-14 09:56:26.658429 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.658434 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.658438 | controller |     - '          exit 1'
2026-03-14 09:56:26.658442 | controller |     - '        fi'
2026-03-14 09:56:26.658446 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.658451 | controller |     - '        '
2026-03-14 09:56:26.658455 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.658459 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.658465 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.658469 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.658473 | controller |       --timeout=300s'
2026-03-14 09:56:26.658477 | controller |     - '        '
2026-03-14 09:56:26.658482 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.658486 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.658490 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.658495 | controller |     - '        '
2026-03-14 09:56:26.658499 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.658503 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.658507 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.658512 | controller |     - '        '
2026-03-14 09:56:26.658522 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.658527 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.658531 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.658535 | controller |       available"'
2026-03-14 09:56:26.658539 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.658544 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.658548 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.658552 | controller |     - '          exit 1'
2026-03-14 09:56:26.658556 | controller |     - '        fi'
2026-03-14 09:56:26.658561 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.658565 | controller |     - '        '
2026-03-14 09:56:26.658569 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.658573 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.658578 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.658582 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.658586 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.658591 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.658607 | controller |       -- \'
2026-03-14 09:56:26.658615 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.658619 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.658623 | controller |     - '    logger.go:42: 09:38:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.658628 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.658632 | controller |     - '    logger.go:42: 09:38:56 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.658636 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.658640 | controller |       found'
2026-03-14 09:56:26.658645 | controller |     - '    logger.go:42: 09:38:57 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.658649 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.658653 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.658659 | controller |     - '        '
2026-03-14 09:56:26.658664 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.658668 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.658672 | controller |     - '        '
2026-03-14 09:56:26.658677 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.658681 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.658685 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.658690 | controller |     - '          exit 1'
2026-03-14 09:56:26.658694 | controller |     - '        fi'
2026-03-14 09:56:26.658698 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.658703 | controller |     - '        '
2026-03-14 09:56:26.658707 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.658711 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.658715 | controller |       | base64 -d)'
2026-03-14 09:56:26.658720 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.658724 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.658728 | controller |     - '          exit 1'
2026-03-14 09:56:26.658733 | controller |     - '        fi'
2026-03-14 09:56:26.658737 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.658741 | controller |     - '        '
2026-03-14 09:56:26.658746 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.658750 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.658754 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.658759 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.658764 | controller |       --timeout=300s'
2026-03-14 09:56:26.658770 | controller |     - '        '
2026-03-14 09:56:26.658777 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.658787 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.658793 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.658798 | controller |     - '        '
2026-03-14 09:56:26.658802 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.658806 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.658811 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.658815 | controller |     - '        '
2026-03-14 09:56:26.658827 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.658832 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.658836 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.658841 | controller |       available"'
2026-03-14 09:56:26.658845 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.658850 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.658854 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.658858 | controller |     - '          exit 1'
2026-03-14 09:56:26.658863 | controller |     - '        fi'
2026-03-14 09:56:26.658867 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.658873 | controller |     - '        '
2026-03-14 09:56:26.658878 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.658885 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.658889 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.658894 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.658898 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.658902 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.658907 | controller |       -- \'
2026-03-14 09:56:26.658911 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.658915 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.658920 | controller |     - '    logger.go:42: 09:38:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.658924 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.658928 | controller |     - '    logger.go:42: 09:38:57 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.658933 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.658937 | controller |       found'
2026-03-14 09:56:26.658941 | controller |     - '    logger.go:42: 09:38:58 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.658945 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.658950 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.658954 | controller |     - '        '
2026-03-14 09:56:26.658959 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.658978 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.658982 | controller |     - '        '
2026-03-14 09:56:26.658986 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.658991 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.658995 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.658999 | controller |     - '          exit 1'
2026-03-14 09:56:26.659004 | controller |     - '        fi'
2026-03-14 09:56:26.659008 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.659013 | controller |     - '        '
2026-03-14 09:56:26.659017 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.659021 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.659026 | controller |       | base64 -d)'
2026-03-14 09:56:26.659032 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.659036 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.659041 | controller |     - '          exit 1'
2026-03-14 09:56:26.659045 | controller |     - '        fi'
2026-03-14 09:56:26.659049 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.659054 | controller |     - '        '
2026-03-14 09:56:26.659058 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.659062 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.659066 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.659071 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.659075 | controller |       --timeout=300s'
2026-03-14 09:56:26.659079 | controller |     - '        '
2026-03-14 09:56:26.659084 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.659088 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.659096 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.659100 | controller |     - '        '
2026-03-14 09:56:26.659104 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.659109 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.659113 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.659117 | controller |     - '        '
2026-03-14 09:56:26.659122 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.659134 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.659139 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.659143 | controller |       available"'
2026-03-14 09:56:26.659148 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.659152 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.659156 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.659161 | controller |     - '          exit 1'
2026-03-14 09:56:26.659165 | controller |     - '        fi'
2026-03-14 09:56:26.659169 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.659174 | controller |     - '        '
2026-03-14 09:56:26.659178 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.659182 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.659187 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.659191 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.659195 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.659200 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.659204 | controller |       -- \'
2026-03-14 09:56:26.659208 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.659215 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.659219 | controller |     - '    logger.go:42: 09:38:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.659223 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.659228 | controller |     - '    logger.go:42: 09:38:58 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.659232 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.659236 | controller |       found'
2026-03-14 09:56:26.659241 | controller |     - '    logger.go:42: 09:38:59 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.659245 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.659249 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.659254 | controller |     - '        '
2026-03-14 09:56:26.659258 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.659262 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.659267 | controller |     - '        '
2026-03-14 09:56:26.659271 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.659275 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.659280 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.659286 | controller |     - '          exit 1'
2026-03-14 09:56:26.659290 | controller |     - '        fi'
2026-03-14 09:56:26.659294 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.659299 | controller |     - '        '
2026-03-14 09:56:26.659303 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.659309 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.659313 | controller |       | base64 -d)'
2026-03-14 09:56:26.659318 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.659322 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.659326 | controller |     - '          exit 1'
2026-03-14 09:56:26.659331 | controller |     - '        fi'
2026-03-14 09:56:26.659335 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.659339 | controller |     - '        '
2026-03-14 09:56:26.659344 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.659348 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.659352 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.659357 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.659361 | controller |       --timeout=300s'
2026-03-14 09:56:26.659365 | controller |     - '        '
2026-03-14 09:56:26.659369 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.659374 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.659378 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.659383 | controller |     - '        '
2026-03-14 09:56:26.659387 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.659391 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.659395 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.659400 | controller |     - '        '
2026-03-14 09:56:26.659404 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.659408 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.659419 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.659424 | controller |       available"'
2026-03-14 09:56:26.659428 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.659433 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.659437 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.659441 | controller |     - '          exit 1'
2026-03-14 09:56:26.659446 | controller |     - '        fi'
2026-03-14 09:56:26.659450 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.659454 | controller |     - '        '
2026-03-14 09:56:26.659459 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.659465 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.659469 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.659474 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.659478 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.659482 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.659488 | controller |       -- \'
2026-03-14 09:56:26.659493 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.659497 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.659502 | controller |     - '    logger.go:42: 09:38:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.659506 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.659512 | controller |     - '    logger.go:42: 09:38:59 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.659518 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.659524 | controller |       found'
2026-03-14 09:56:26.659530 | controller |     - '    logger.go:42: 09:39:00 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.659536 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.659545 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.659549 | controller |     - '        '
2026-03-14 09:56:26.659554 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.659558 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.659563 | controller |     - '        '
2026-03-14 09:56:26.659567 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.659571 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.659576 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.659580 | controller |     - '          exit 1'
2026-03-14 09:56:26.659584 | controller |     - '        fi'
2026-03-14 09:56:26.659589 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.659593 | controller |     - '        '
2026-03-14 09:56:26.659612 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.659617 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.659621 | controller |       | base64 -d)'
2026-03-14 09:56:26.659625 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.659630 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.659634 | controller |     - '          exit 1'
2026-03-14 09:56:26.659638 | controller |     - '        fi'
2026-03-14 09:56:26.659642 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.659647 | controller |     - '        '
2026-03-14 09:56:26.659651 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.659655 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.659660 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.659664 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.659668 | controller |       --timeout=300s'
2026-03-14 09:56:26.659672 | controller |     - '        '
2026-03-14 09:56:26.659677 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.659681 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.659685 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.659690 | controller |     - '        '
2026-03-14 09:56:26.659694 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.659698 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.659702 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.659709 | controller |     - '        '
2026-03-14 09:56:26.659713 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.659718 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.659730 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.659735 | controller |       available"'
2026-03-14 09:56:26.659739 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.659743 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.659748 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.659752 | controller |     - '          exit 1'
2026-03-14 09:56:26.659756 | controller |     - '        fi'
2026-03-14 09:56:26.659761 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.659767 | controller |     - '        '
2026-03-14 09:56:26.659773 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.659779 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.659785 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.659792 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.659796 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.659800 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.659804 | controller |       -- \'
2026-03-14 09:56:26.659809 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.659813 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.659817 | controller |     - '    logger.go:42: 09:39:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.659822 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.659826 | controller |     - '    logger.go:42: 09:39:01 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.659830 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.659834 | controller |       found'
2026-03-14 09:56:26.659839 | controller |     - '    logger.go:42: 09:39:02 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.659843 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.659847 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.659851 | controller |     - '        '
2026-03-14 09:56:26.659856 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.659860 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.659864 | controller |     - '        '
2026-03-14 09:56:26.659868 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.659872 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.659877 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.659881 | controller |     - '          exit 1'
2026-03-14 09:56:26.659885 | controller |     - '        fi'
2026-03-14 09:56:26.659889 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.659894 | controller |     - '        '
2026-03-14 09:56:26.659898 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.659902 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.659906 | controller |       | base64 -d)'
2026-03-14 09:56:26.659911 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.659917 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.659922 | controller |     - '          exit 1'
2026-03-14 09:56:26.659926 | controller |     - '        fi'
2026-03-14 09:56:26.659930 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.659934 | controller |     - '        '
2026-03-14 09:56:26.659939 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.659943 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.659947 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.659951 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.659956 | controller |       --timeout=300s'
2026-03-14 09:56:26.659960 | controller |     - '        '
2026-03-14 09:56:26.659964 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.659968 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.659973 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.659977 | controller |     - '        '
2026-03-14 09:56:26.659981 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.659985 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.659990 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.659994 | controller |     - '        '
2026-03-14 09:56:26.659998 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.660002 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.660007 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.660021 | controller |       available"'
2026-03-14 09:56:26.660026 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.660030 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.660034 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.660039 | controller |     - '          exit 1'
2026-03-14 09:56:26.660043 | controller |     - '        fi'
2026-03-14 09:56:26.660047 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.660051 | controller |     - '        '
2026-03-14 09:56:26.660056 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.660060 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.660066 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.660086 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.660094 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.660103 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.660109 | controller |       -- \'
2026-03-14 09:56:26.660115 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.660121 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.660126 | controller |     - '    logger.go:42: 09:39:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.660132 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.660142 | controller |     - '    logger.go:42: 09:39:02 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.660148 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.660153 | controller |       found'
2026-03-14 09:56:26.660159 | controller |     - '    logger.go:42: 09:39:03 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.660165 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.660170 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.660176 | controller |     - '        '
2026-03-14 09:56:26.660182 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.660188 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.660195 | controller |     - '        '
2026-03-14 09:56:26.660200 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.660204 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.660209 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.660213 | controller |     - '          exit 1'
2026-03-14 09:56:26.660217 | controller |     - '        fi'
2026-03-14 09:56:26.660222 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.660229 | controller |     - '        '
2026-03-14 09:56:26.660233 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.660237 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.660243 | controller |       | base64 -d)'
2026-03-14 09:56:26.660247 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.660252 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.660256 | controller |     - '          exit 1'
2026-03-14 09:56:26.660260 | controller |     - '        fi'
2026-03-14 09:56:26.660264 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.660268 | controller |     - '        '
2026-03-14 09:56:26.660273 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.660277 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.660281 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.660285 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.660290 | controller |       --timeout=300s'
2026-03-14 09:56:26.660294 | controller |     - '        '
2026-03-14 09:56:26.660298 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.660302 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.660307 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.660311 | controller |     - '        '
2026-03-14 09:56:26.660315 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.660319 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.660324 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.660368 | controller |     - '        '
2026-03-14 09:56:26.660372 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.660376 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.660381 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.660385 | controller |       available"'
2026-03-14 09:56:26.660399 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.660408 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.660412 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.660417 | controller |     - '          exit 1'
2026-03-14 09:56:26.660421 | controller |     - '        fi'
2026-03-14 09:56:26.660425 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.660429 | controller |     - '        '
2026-03-14 09:56:26.660434 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.660438 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.660442 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.660446 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.660450 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.660455 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.660459 | controller |       -- \'
2026-03-14 09:56:26.660463 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.660467 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.660472 | controller |     - '    logger.go:42: 09:39:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.660476 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.660480 | controller |     - '    logger.go:42: 09:39:03 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.660484 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.660489 | controller |       found'
2026-03-14 09:56:26.660493 | controller |     - '    logger.go:42: 09:39:04 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.660497 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.660501 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.660505 | controller |     - '        '
2026-03-14 09:56:26.660510 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.660514 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.660518 | controller |     - '        '
2026-03-14 09:56:26.660522 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.660527 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.660531 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.660535 | controller |     - '          exit 1'
2026-03-14 09:56:26.660539 | controller |     - '        fi'
2026-03-14 09:56:26.660545 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.660550 | controller |     - '        '
2026-03-14 09:56:26.660554 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.660558 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.660563 | controller |       | base64 -d)'
2026-03-14 09:56:26.660567 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.660571 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.660576 | controller |     - '          exit 1'
2026-03-14 09:56:26.660580 | controller |     - '        fi'
2026-03-14 09:56:26.660584 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.660589 | controller |     - '        '
2026-03-14 09:56:26.660593 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.660615 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.660620 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.660631 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.660636 | controller |       --timeout=300s'
2026-03-14 09:56:26.660640 | controller |     - '        '
2026-03-14 09:56:26.660644 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.660649 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.660653 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.660657 | controller |     - '        '
2026-03-14 09:56:26.660662 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.660666 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.660670 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.660674 | controller |     - '        '
2026-03-14 09:56:26.660679 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.660683 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.660687 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.660692 | controller |       available"'
2026-03-14 09:56:26.660696 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.660707 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.660714 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.660719 | controller |     - '          exit 1'
2026-03-14 09:56:26.660723 | controller |     - '        fi'
2026-03-14 09:56:26.660728 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.660732 | controller |     - '        '
2026-03-14 09:56:26.660736 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.660741 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.660745 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.660749 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.660754 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.660758 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.660763 | controller |       -- \'
2026-03-14 09:56:26.660769 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.660784 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.660799 | controller |     - '    logger.go:42: 09:39:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.660806 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.660811 | controller |     - '    logger.go:42: 09:39:04 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.660815 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.660820 | controller |       found'
2026-03-14 09:56:26.660824 | controller |     - '    logger.go:42: 09:39:05 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.660829 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.660833 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.660837 | controller |     - '        '
2026-03-14 09:56:26.660842 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.660849 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.660853 | controller |     - '        '
2026-03-14 09:56:26.660858 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.660862 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.660866 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.660871 | controller |     - '          exit 1'
2026-03-14 09:56:26.660875 | controller |     - '        fi'
2026-03-14 09:56:26.660879 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.660884 | controller |     - '        '
2026-03-14 09:56:26.660888 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.660892 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.660897 | controller |       | base64 -d)'
2026-03-14 09:56:26.660901 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.660905 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.660910 | controller |     - '          exit 1'
2026-03-14 09:56:26.660914 | controller |     - '        fi'
2026-03-14 09:56:26.660918 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.660922 | controller |     - '        '
2026-03-14 09:56:26.660927 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.660931 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.660935 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.660940 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.660944 | controller |       --timeout=300s'
2026-03-14 09:56:26.660948 | controller |     - '        '
2026-03-14 09:56:26.660952 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.660957 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.660961 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.660965 | controller |     - '        '
2026-03-14 09:56:26.660970 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.660974 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.660978 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.660983 | controller |     - '        '
2026-03-14 09:56:26.660987 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.660991 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.660995 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.661000 | controller |       available"'
2026-03-14 09:56:26.661004 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.661008 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.661024 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.661029 | controller |     - '          exit 1'
2026-03-14 09:56:26.661033 | controller |     - '        fi'
2026-03-14 09:56:26.661038 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.661042 | controller |     - '        '
2026-03-14 09:56:26.661046 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.661051 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.661057 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.661061 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.661066 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.661070 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.661074 | controller |       -- \'
2026-03-14 09:56:26.661079 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.661083 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.661087 | controller |     - '    logger.go:42: 09:39:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.661091 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.661096 | controller |     - '    logger.go:42: 09:39:05 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.661100 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.661104 | controller |       found'
2026-03-14 09:56:26.661108 | controller |     - '    logger.go:42: 09:39:06 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.661113 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.661117 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.661121 | controller |     - '        '
2026-03-14 09:56:26.661126 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.661130 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.661134 | controller |     - '        '
2026-03-14 09:56:26.661138 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.661143 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.661147 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.661151 | controller |     - '          exit 1'
2026-03-14 09:56:26.661156 | controller |     - '        fi'
2026-03-14 09:56:26.661160 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.661164 | controller |     - '        '
2026-03-14 09:56:26.661168 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.661173 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.661177 | controller |       | base64 -d)'
2026-03-14 09:56:26.661181 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.661186 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.661190 | controller |     - '          exit 1'
2026-03-14 09:56:26.661194 | controller |     - '        fi'
2026-03-14 09:56:26.661198 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.661203 | controller |     - '        '
2026-03-14 09:56:26.661207 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.661211 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.661215 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.661220 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.661224 | controller |       --timeout=300s'
2026-03-14 09:56:26.661228 | controller |     - '        '
2026-03-14 09:56:26.661232 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.661237 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.661243 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.661265 | controller |     - '        '
2026-03-14 09:56:26.661269 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.661273 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.661278 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.661285 | controller |     - '        '
2026-03-14 09:56:26.661289 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.661293 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.661298 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.661302 | controller |       available"'
2026-03-14 09:56:26.661306 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.661311 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.661322 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.661326 | controller |     - '          exit 1'
2026-03-14 09:56:26.661331 | controller |     - '        fi'
2026-03-14 09:56:26.661335 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.661339 | controller |     - '        '
2026-03-14 09:56:26.661344 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.661348 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.661352 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.661357 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.661361 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.661365 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.661370 | controller |       -- \'
2026-03-14 09:56:26.661374 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.661378 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.661383 | controller |     - '    logger.go:42: 09:39:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.661387 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.661391 | controller |     - '    logger.go:42: 09:39:06 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.661397 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.661402 | controller |       found'
2026-03-14 09:56:26.661406 | controller |     - '    logger.go:42: 09:39:07 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.661411 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.661415 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.661419 | controller |     - '        '
2026-03-14 09:56:26.661423 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.661428 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.661432 | controller |     - '        '
2026-03-14 09:56:26.661436 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.661441 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.661445 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.661450 | controller |     - '          exit 1'
2026-03-14 09:56:26.661455 | controller |     - '        fi'
2026-03-14 09:56:26.661460 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.661464 | controller |     - '        '
2026-03-14 09:56:26.661468 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.661473 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.661477 | controller |       | base64 -d)'
2026-03-14 09:56:26.661482 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.661486 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.661490 | controller |     - '          exit 1'
2026-03-14 09:56:26.661494 | controller |     - '        fi'
2026-03-14 09:56:26.661499 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.661503 | controller |     - '        '
2026-03-14 09:56:26.661507 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.661512 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.661518 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.661524 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.661530 | controller |       --timeout=300s'
2026-03-14 09:56:26.661536 | controller |     - '        '
2026-03-14 09:56:26.661542 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.661548 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.661556 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.661562 | controller |     - '        '
2026-03-14 09:56:26.661566 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.661570 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.661575 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.661579 | controller |     - '        '
2026-03-14 09:56:26.661583 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.661588 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.661592 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.661671 | controller |       available"'
2026-03-14 09:56:26.661676 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.661681 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.661685 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.661690 | controller |     - '          exit 1'
2026-03-14 09:56:26.661694 | controller |     - '        fi'
2026-03-14 09:56:26.661707 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.661712 | controller |     - '        '
2026-03-14 09:56:26.661717 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.661721 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.661725 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.661730 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.661734 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.661739 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.661749 | controller |       -- \'
2026-03-14 09:56:26.661754 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.661758 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.661764 | controller |     - '    logger.go:42: 09:39:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.661770 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.661776 | controller |     - '    logger.go:42: 09:39:07 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.661783 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.661789 | controller |       found'
2026-03-14 09:56:26.661795 | controller |     - '    logger.go:42: 09:39:08 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.661800 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.661804 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.661809 | controller |     - '        '
2026-03-14 09:56:26.661813 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.661817 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.661821 | controller |     - '        '
2026-03-14 09:56:26.661826 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.661830 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.661834 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.661838 | controller |     - '          exit 1'
2026-03-14 09:56:26.661843 | controller |     - '        fi'
2026-03-14 09:56:26.661847 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.661851 | controller |     - '        '
2026-03-14 09:56:26.661855 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.661860 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.661864 | controller |       | base64 -d)'
2026-03-14 09:56:26.661868 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.661872 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.661877 | controller |     - '          exit 1'
2026-03-14 09:56:26.661881 | controller |     - '        fi'
2026-03-14 09:56:26.661885 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.661889 | controller |     - '        '
2026-03-14 09:56:26.661894 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.661898 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.661902 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.661906 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.661910 | controller |       --timeout=300s'
2026-03-14 09:56:26.661915 | controller |     - '        '
2026-03-14 09:56:26.661919 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.661923 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.661927 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.661932 | controller |     - '        '
2026-03-14 09:56:26.661936 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.661940 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.661944 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.661951 | controller |     - '        '
2026-03-14 09:56:26.661955 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.661959 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.661964 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.661968 | controller |       available"'
2026-03-14 09:56:26.661972 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.661976 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.661981 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.661985 | controller |     - '          exit 1'
2026-03-14 09:56:26.661989 | controller |     - '        fi'
2026-03-14 09:56:26.662001 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.662006 | controller |     - '        '
2026-03-14 09:56:26.662010 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.662014 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.662019 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.662023 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.662027 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.662031 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.662036 | controller |       -- \'
2026-03-14 09:56:26.662040 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.662044 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.662049 | controller |     - '    logger.go:42: 09:39:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.662053 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.662057 | controller |     - '    logger.go:42: 09:39:09 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.662061 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.662081 | controller |       found'
2026-03-14 09:56:26.662086 | controller |     - '    logger.go:42: 09:39:10 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.662090 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.662094 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.662099 | controller |     - '        '
2026-03-14 09:56:26.662103 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.662107 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.662111 | controller |     - '        '
2026-03-14 09:56:26.662115 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.662120 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.662124 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.662128 | controller |     - '          exit 1'
2026-03-14 09:56:26.662132 | controller |     - '        fi'
2026-03-14 09:56:26.662137 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.662141 | controller |     - '        '
2026-03-14 09:56:26.662145 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.662151 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.662156 | controller |       | base64 -d)'
2026-03-14 09:56:26.662160 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.662164 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.662171 | controller |     - '          exit 1'
2026-03-14 09:56:26.662175 | controller |     - '        fi'
2026-03-14 09:56:26.662179 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.662183 | controller |     - '        '
2026-03-14 09:56:26.662188 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.662192 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.662196 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.662200 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.662204 | controller |       --timeout=300s'
2026-03-14 09:56:26.662209 | controller |     - '        '
2026-03-14 09:56:26.662213 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.662217 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.662221 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.662226 | controller |     - '        '
2026-03-14 09:56:26.662230 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.662234 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.662238 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.662242 | controller |     - '        '
2026-03-14 09:56:26.662247 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.662251 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.662255 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.662259 | controller |       available"'
2026-03-14 09:56:26.662264 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.662268 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.662272 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.662276 | controller |     - '          exit 1'
2026-03-14 09:56:26.662280 | controller |     - '        fi'
2026-03-14 09:56:26.662285 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.662289 | controller |     - '        '
2026-03-14 09:56:26.662300 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.662304 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.662309 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.662313 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.662318 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.662322 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.662326 | controller |       -- \'
2026-03-14 09:56:26.662330 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.662335 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.662341 | controller |     - '    logger.go:42: 09:39:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.662346 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.662350 | controller |     - '    logger.go:42: 09:39:10 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.662357 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.662363 | controller |       found'
2026-03-14 09:56:26.662367 | controller |     - '    logger.go:42: 09:39:11 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.662371 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.662375 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.662380 | controller |     - '        '
2026-03-14 09:56:26.662384 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.662388 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.662393 | controller |     - '        '
2026-03-14 09:56:26.662397 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.662401 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.662406 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.662410 | controller |     - '          exit 1'
2026-03-14 09:56:26.662416 | controller |     - '        fi'
2026-03-14 09:56:26.662422 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.662428 | controller |     - '        '
2026-03-14 09:56:26.662433 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.662440 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.662446 | controller |       | base64 -d)'
2026-03-14 09:56:26.662451 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.662457 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.662463 | controller |     - '          exit 1'
2026-03-14 09:56:26.662468 | controller |     - '        fi'
2026-03-14 09:56:26.662476 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.662482 | controller |     - '        '
2026-03-14 09:56:26.662487 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.662492 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.662496 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.662500 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.662505 | controller |       --timeout=300s'
2026-03-14 09:56:26.662509 | controller |     - '        '
2026-03-14 09:56:26.662513 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.662518 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.662522 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.662526 | controller |     - '        '
2026-03-14 09:56:26.662531 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.662535 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.662539 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.662544 | controller |     - '        '
2026-03-14 09:56:26.662548 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.662552 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.662556 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.662561 | controller |       available"'
2026-03-14 09:56:26.662565 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.662569 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.662576 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.662580 | controller |     - '          exit 1'
2026-03-14 09:56:26.662584 | controller |     - '        fi'
2026-03-14 09:56:26.662589 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.662593 | controller |     - '        '
2026-03-14 09:56:26.662635 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.662640 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.662645 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.662652 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.662656 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.662660 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.662665 | controller |       -- \'
2026-03-14 09:56:26.662669 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.662673 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.662677 | controller |     - '    logger.go:42: 09:39:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.662682 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.662686 | controller |     - '    logger.go:42: 09:39:11 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.662690 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.662695 | controller |       found'
2026-03-14 09:56:26.662699 | controller |     - '    logger.go:42: 09:39:12 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.662705 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.662709 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.662713 | controller |     - '        '
2026-03-14 09:56:26.662718 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.662722 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.662726 | controller |     - '        '
2026-03-14 09:56:26.662730 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.662735 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.662739 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.662743 | controller |     - '          exit 1'
2026-03-14 09:56:26.662747 | controller |     - '        fi'
2026-03-14 09:56:26.662752 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.662756 | controller |     - '        '
2026-03-14 09:56:26.662760 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.662766 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.662772 | controller |       | base64 -d)'
2026-03-14 09:56:26.662778 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.662784 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.662791 | controller |     - '          exit 1'
2026-03-14 09:56:26.662796 | controller |     - '        fi'
2026-03-14 09:56:26.662800 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.662805 | controller |     - '        '
2026-03-14 09:56:26.662809 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.662813 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.662817 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.662824 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.662828 | controller |       --timeout=300s'
2026-03-14 09:56:26.662833 | controller |     - '        '
2026-03-14 09:56:26.662837 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.662841 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.662846 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.662850 | controller |     - '        '
2026-03-14 09:56:26.662854 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.662858 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.662863 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.662867 | controller |     - '        '
2026-03-14 09:56:26.662871 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.662875 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.662880 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.662884 | controller |       available"'
2026-03-14 09:56:26.662888 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.662892 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.662897 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.662901 | controller |     - '          exit 1'
2026-03-14 09:56:26.662905 | controller |     - '        fi'
2026-03-14 09:56:26.662909 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.662913 | controller |     - '        '
2026-03-14 09:56:26.662918 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.662930 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.662935 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.662939 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.662944 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.662948 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.662952 | controller |       -- \'
2026-03-14 09:56:26.662957 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.662980 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.662985 | controller |     - '    logger.go:42: 09:39:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.662990 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.662994 | controller |     - '    logger.go:42: 09:39:12 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.662998 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.663002 | controller |       found'
2026-03-14 09:56:26.663007 | controller |     - '    logger.go:42: 09:39:13 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.663011 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.663015 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.663019 | controller |     - '        '
2026-03-14 09:56:26.663024 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.663030 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.663034 | controller |     - '        '
2026-03-14 09:56:26.663038 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.663043 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.663047 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.663051 | controller |     - '          exit 1'
2026-03-14 09:56:26.663055 | controller |     - '        fi'
2026-03-14 09:56:26.663060 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.663064 | controller |     - '        '
2026-03-14 09:56:26.663068 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.663072 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.663077 | controller |       | base64 -d)'
2026-03-14 09:56:26.663081 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.663085 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.663089 | controller |     - '          exit 1'
2026-03-14 09:56:26.663093 | controller |     - '        fi'
2026-03-14 09:56:26.663098 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.663102 | controller |     - '        '
2026-03-14 09:56:26.663106 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.663110 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.663115 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.663119 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.663123 | controller |       --timeout=300s'
2026-03-14 09:56:26.663127 | controller |     - '        '
2026-03-14 09:56:26.663132 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.663138 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.663143 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.663147 | controller |     - '        '
2026-03-14 09:56:26.663151 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.663156 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.663160 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.663164 | controller |     - '        '
2026-03-14 09:56:26.663168 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.663173 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.663177 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.663181 | controller |       available"'
2026-03-14 09:56:26.663185 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.663190 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.663194 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.663198 | controller |     - '          exit 1'
2026-03-14 09:56:26.663202 | controller |     - '        fi'
2026-03-14 09:56:26.663207 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.663211 | controller |     - '        '
2026-03-14 09:56:26.663215 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.663226 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.663232 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.663237 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.663241 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.663246 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.663250 | controller |       -- \'
2026-03-14 09:56:26.663254 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.663259 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.663263 | controller |     - '    logger.go:42: 09:39:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.663267 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.663272 | controller |     - '    logger.go:42: 09:39:13 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.663276 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.663280 | controller |       found'
2026-03-14 09:56:26.663285 | controller |     - '    logger.go:42: 09:39:14 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.663289 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.663293 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.663298 | controller |     - '        '
2026-03-14 09:56:26.663302 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.663310 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.663314 | controller |     - '        '
2026-03-14 09:56:26.663318 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.663323 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.663329 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.663333 | controller |     - '          exit 1'
2026-03-14 09:56:26.663338 | controller |     - '        fi'
2026-03-14 09:56:26.663342 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.663346 | controller |     - '        '
2026-03-14 09:56:26.663351 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.663355 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.663359 | controller |       | base64 -d)'
2026-03-14 09:56:26.663364 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.663368 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.663372 | controller |     - '          exit 1'
2026-03-14 09:56:26.663377 | controller |     - '        fi'
2026-03-14 09:56:26.663381 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.663385 | controller |     - '        '
2026-03-14 09:56:26.663390 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.663394 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.663399 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.663403 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.663407 | controller |       --timeout=300s'
2026-03-14 09:56:26.663412 | controller |     - '        '
2026-03-14 09:56:26.663416 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.663420 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.663425 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.663431 | controller |     - '        '
2026-03-14 09:56:26.663435 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.663440 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.663444 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.663448 | controller |     - '        '
2026-03-14 09:56:26.663453 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.663457 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.663461 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.663466 | controller |       available"'
2026-03-14 09:56:26.663470 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.663474 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.663482 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.663486 | controller |     - '          exit 1'
2026-03-14 09:56:26.663491 | controller |     - '        fi'
2026-03-14 09:56:26.663495 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.663499 | controller |     - '        '
2026-03-14 09:56:26.663504 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.663508 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.663519 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.663524 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.663528 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.663533 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.663537 | controller |       -- \'
2026-03-14 09:56:26.663542 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.663546 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.663550 | controller |     - '    logger.go:42: 09:39:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.663555 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.663559 | controller |     - '    logger.go:42: 09:39:15 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.663564 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.663568 | controller |       found'
2026-03-14 09:56:26.663572 | controller |     - '    logger.go:42: 09:39:16 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.663577 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.663581 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.663585 | controller |     - '        '
2026-03-14 09:56:26.663590 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.663605 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.663612 | controller |     - '        '
2026-03-14 09:56:26.663629 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.663636 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.663642 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.663648 | controller |     - '          exit 1'
2026-03-14 09:56:26.663657 | controller |     - '        fi'
2026-03-14 09:56:26.663661 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.663668 | controller |     - '        '
2026-03-14 09:56:26.663672 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.663677 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.663681 | controller |       | base64 -d)'
2026-03-14 09:56:26.663685 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.663690 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.663694 | controller |     - '          exit 1'
2026-03-14 09:56:26.663698 | controller |     - '        fi'
2026-03-14 09:56:26.663703 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.663707 | controller |     - '        '
2026-03-14 09:56:26.663711 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.663716 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.663720 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.663724 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.663729 | controller |       --timeout=300s'
2026-03-14 09:56:26.663733 | controller |     - '        '
2026-03-14 09:56:26.663737 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.663742 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.663746 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.663752 | controller |     - '        '
2026-03-14 09:56:26.663757 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.663762 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.663768 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.663774 | controller |     - '        '
2026-03-14 09:56:26.663781 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.663787 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.663794 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.663799 | controller |       available"'
2026-03-14 09:56:26.663803 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.663808 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.663812 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.663817 | controller |     - '          exit 1'
2026-03-14 09:56:26.663821 | controller |     - '        fi'
2026-03-14 09:56:26.663825 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.663830 | controller |     - '        '
2026-03-14 09:56:26.663834 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.663838 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.663852 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.663857 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.663862 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.663866 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.663871 | controller |       -- \'
2026-03-14 09:56:26.663878 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.663882 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.663886 | controller |     - '    logger.go:42: 09:39:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.663891 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.663895 | controller |     - '    logger.go:42: 09:39:16 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.663899 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.663904 | controller |       found'
2026-03-14 09:56:26.663908 | controller |     - '    logger.go:42: 09:39:17 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.663912 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.663918 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.663922 | controller |     - '        '
2026-03-14 09:56:26.663926 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.663931 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.663938 | controller |     - '        '
2026-03-14 09:56:26.663942 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.663946 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.663951 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.663955 | controller |     - '          exit 1'
2026-03-14 09:56:26.663959 | controller |     - '        fi'
2026-03-14 09:56:26.663964 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.663968 | controller |     - '        '
2026-03-14 09:56:26.663972 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.663977 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.663981 | controller |       | base64 -d)'
2026-03-14 09:56:26.663985 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.663989 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.663994 | controller |     - '          exit 1'
2026-03-14 09:56:26.663998 | controller |     - '        fi'
2026-03-14 09:56:26.664002 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.664007 | controller |     - '        '
2026-03-14 09:56:26.664011 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.664015 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.664019 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.664024 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.664028 | controller |       --timeout=300s'
2026-03-14 09:56:26.664032 | controller |     - '        '
2026-03-14 09:56:26.664037 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.664041 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.664045 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.664049 | controller |     - '        '
2026-03-14 09:56:26.664054 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.664058 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.664062 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.664067 | controller |     - '        '
2026-03-14 09:56:26.664071 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.664077 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.664081 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.664086 | controller |       available"'
2026-03-14 09:56:26.664090 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.664094 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.664099 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.664103 | controller |     - '          exit 1'
2026-03-14 09:56:26.664107 | controller |     - '        fi'
2026-03-14 09:56:26.664111 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.664116 | controller |     - '        '
2026-03-14 09:56:26.664120 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.664124 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.664129 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.664139 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.664144 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.664148 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.664153 | controller |       -- \'
2026-03-14 09:56:26.664157 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.664162 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.664166 | controller |     - '    logger.go:42: 09:39:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.664170 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.664175 | controller |     - '    logger.go:42: 09:39:17 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.664179 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.664183 | controller |       found'
2026-03-14 09:56:26.664187 | controller |     - '    logger.go:42: 09:39:18 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.664192 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.664196 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.664200 | controller |     - '        '
2026-03-14 09:56:26.664205 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.664209 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.664213 | controller |     - '        '
2026-03-14 09:56:26.664218 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.664222 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.664226 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.664230 | controller |     - '          exit 1'
2026-03-14 09:56:26.664235 | controller |     - '        fi'
2026-03-14 09:56:26.664239 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.664243 | controller |     - '        '
2026-03-14 09:56:26.664248 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.664252 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.664256 | controller |       | base64 -d)'
2026-03-14 09:56:26.664261 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.664265 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.664269 | controller |     - '          exit 1'
2026-03-14 09:56:26.664275 | controller |     - '        fi'
2026-03-14 09:56:26.664280 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.664284 | controller |     - '        '
2026-03-14 09:56:26.664288 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.664293 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.664297 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.664301 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.664306 | controller |       --timeout=300s'
2026-03-14 09:56:26.664310 | controller |     - '        '
2026-03-14 09:56:26.664314 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.664319 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.664323 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.664327 | controller |     - '        '
2026-03-14 09:56:26.664332 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.664336 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.664340 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.664345 | controller |     - '        '
2026-03-14 09:56:26.664349 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.664353 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.664357 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.664362 | controller |       available"'
2026-03-14 09:56:26.664366 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.664370 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.664375 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.664379 | controller |     - '          exit 1'
2026-03-14 09:56:26.664383 | controller |     - '        fi'
2026-03-14 09:56:26.664388 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.664392 | controller |     - '        '
2026-03-14 09:56:26.664396 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.664400 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.664405 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.664415 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.664420 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.664424 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.664428 | controller |       -- \'
2026-03-14 09:56:26.664433 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.664437 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.664441 | controller |     - '    logger.go:42: 09:39:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.664446 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.664450 | controller |     - '    logger.go:42: 09:39:18 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.664454 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.664460 | controller |       found'
2026-03-14 09:56:26.664467 | controller |     - '    logger.go:42: 09:39:19 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.664472 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.664476 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.664481 | controller |     - '        '
2026-03-14 09:56:26.664485 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.664489 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.664493 | controller |     - '        '
2026-03-14 09:56:26.664498 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.664504 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.664508 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.664512 | controller |     - '          exit 1'
2026-03-14 09:56:26.664517 | controller |     - '        fi'
2026-03-14 09:56:26.664521 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.664527 | controller |     - '        '
2026-03-14 09:56:26.664531 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.664535 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.664541 | controller |       | base64 -d)'
2026-03-14 09:56:26.664545 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.664550 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.664554 | controller |     - '          exit 1'
2026-03-14 09:56:26.664558 | controller |     - '        fi'
2026-03-14 09:56:26.664562 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.664567 | controller |     - '        '
2026-03-14 09:56:26.664571 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.664575 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.664579 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.664584 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.664588 | controller |       --timeout=300s'
2026-03-14 09:56:26.664592 | controller |     - '        '
2026-03-14 09:56:26.664610 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.664615 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.664619 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.664623 | controller |     - '        '
2026-03-14 09:56:26.664628 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.664632 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.664636 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.664641 | controller |     - '        '
2026-03-14 09:56:26.664645 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.664649 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.664654 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.664658 | controller |       available"'
2026-03-14 09:56:26.664662 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.664667 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.664671 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.664678 | controller |     - '          exit 1'
2026-03-14 09:56:26.664685 | controller |     - '        fi'
2026-03-14 09:56:26.664689 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.664693 | controller |     - '        '
2026-03-14 09:56:26.664698 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.664702 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.664706 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.664711 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.664722 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.664730 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.664735 | controller |       -- \'
2026-03-14 09:56:26.664739 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.664743 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.664748 | controller |     - '    logger.go:42: 09:39:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.664752 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.664756 | controller |     - '    logger.go:42: 09:39:19 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.664761 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.664767 | controller |       found'
2026-03-14 09:56:26.664773 | controller |     - '    logger.go:42: 09:39:20 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.664779 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.664785 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.664791 | controller |     - '        '
2026-03-14 09:56:26.664796 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.664800 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.664804 | controller |     - '        '
2026-03-14 09:56:26.664808 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.664812 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.664817 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.664821 | controller |     - '          exit 1'
2026-03-14 09:56:26.664825 | controller |     - '        fi'
2026-03-14 09:56:26.664830 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.664834 | controller |     - '        '
2026-03-14 09:56:26.664838 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.664845 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.664849 | controller |       | base64 -d)'
2026-03-14 09:56:26.664853 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.664857 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.664861 | controller |     - '          exit 1'
2026-03-14 09:56:26.664866 | controller |     - '        fi'
2026-03-14 09:56:26.664870 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.664874 | controller |     - '        '
2026-03-14 09:56:26.664879 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.664883 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.664887 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.664891 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.664898 | controller |       --timeout=300s'
2026-03-14 09:56:26.664902 | controller |     - '        '
2026-03-14 09:56:26.664906 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.664911 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.664915 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.664919 | controller |     - '        '
2026-03-14 09:56:26.664924 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.664928 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.664932 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.664937 | controller |     - '        '
2026-03-14 09:56:26.664941 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.664945 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.664950 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.664954 | controller |       available"'
2026-03-14 09:56:26.664958 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.664963 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.664967 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.664971 | controller |     - '          exit 1'
2026-03-14 09:56:26.664975 | controller |     - '        fi'
2026-03-14 09:56:26.664980 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.664984 | controller |     - '        '
2026-03-14 09:56:26.664988 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.664993 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.664997 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.665001 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.665013 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.665018 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.665022 | controller |       -- \'
2026-03-14 09:56:26.665027 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.665031 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.665036 | controller |     - '    logger.go:42: 09:39:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.665040 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.665044 | controller |     - '    logger.go:42: 09:39:20 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.665048 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.665053 | controller |       found'
2026-03-14 09:56:26.665057 | controller |     - '    logger.go:42: 09:39:21 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.665061 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.665066 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.665070 | controller |     - '        '
2026-03-14 09:56:26.665074 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.665079 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.665085 | controller |     - '        '
2026-03-14 09:56:26.665089 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.665093 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.665098 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.665102 | controller |     - '          exit 1'
2026-03-14 09:56:26.665106 | controller |     - '        fi'
2026-03-14 09:56:26.665111 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.665115 | controller |     - '        '
2026-03-14 09:56:26.665119 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.665124 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.665128 | controller |       | base64 -d)'
2026-03-14 09:56:26.665132 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.665137 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.665141 | controller |     - '          exit 1'
2026-03-14 09:56:26.665145 | controller |     - '        fi'
2026-03-14 09:56:26.665150 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.665154 | controller |     - '        '
2026-03-14 09:56:26.665158 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.665163 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.665169 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.665173 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.665177 | controller |       --timeout=300s'
2026-03-14 09:56:26.665182 | controller |     - '        '
2026-03-14 09:56:26.665186 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.665190 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.665194 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.665199 | controller |     - '        '
2026-03-14 09:56:26.665203 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.665207 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.665212 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.665216 | controller |     - '        '
2026-03-14 09:56:26.665220 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.665225 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.665229 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.665233 | controller |       available"'
2026-03-14 09:56:26.665238 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.665242 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.665246 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.665250 | controller |     - '          exit 1'
2026-03-14 09:56:26.665255 | controller |     - '        fi'
2026-03-14 09:56:26.665259 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.665263 | controller |     - '        '
2026-03-14 09:56:26.665268 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.665272 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.665276 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.665283 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.665288 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.665299 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.665303 | controller |       -- \'
2026-03-14 09:56:26.665308 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.665312 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.665317 | controller |     - '    logger.go:42: 09:39:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.665321 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.665325 | controller |     - '    logger.go:42: 09:39:21 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.665330 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.665334 | controller |       found'
2026-03-14 09:56:26.665338 | controller |     - '    logger.go:42: 09:39:22 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.665343 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.665347 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.665352 | controller |     - '        '
2026-03-14 09:56:26.665357 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.665362 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.665367 | controller |     - '        '
2026-03-14 09:56:26.665372 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.665377 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.665382 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.665386 | controller |     - '          exit 1'
2026-03-14 09:56:26.665391 | controller |     - '        fi'
2026-03-14 09:56:26.665395 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.665400 | controller |     - '        '
2026-03-14 09:56:26.665404 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.665408 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.665412 | controller |       | base64 -d)'
2026-03-14 09:56:26.665417 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.665421 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.665425 | controller |     - '          exit 1'
2026-03-14 09:56:26.665430 | controller |     - '        fi'
2026-03-14 09:56:26.665434 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.665438 | controller |     - '        '
2026-03-14 09:56:26.665443 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.665447 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.665451 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.665456 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.665460 | controller |       --timeout=300s'
2026-03-14 09:56:26.665464 | controller |     - '        '
2026-03-14 09:56:26.665468 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.665473 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.665477 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.665483 | controller |     - '        '
2026-03-14 09:56:26.665488 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.665492 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.665496 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.665501 | controller |     - '        '
2026-03-14 09:56:26.665505 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.665509 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.665514 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.665518 | controller |       available"'
2026-03-14 09:56:26.665522 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.665527 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.665531 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.665537 | controller |     - '          exit 1'
2026-03-14 09:56:26.665542 | controller |     - '        fi'
2026-03-14 09:56:26.665546 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.665551 | controller |     - '        '
2026-03-14 09:56:26.665555 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.665559 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.665564 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.665568 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.665572 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.665583 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.665587 | controller |       -- \'
2026-03-14 09:56:26.665592 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.665609 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.665614 | controller |     - '    logger.go:42: 09:39:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.665618 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.665623 | controller |     - '    logger.go:42: 09:39:23 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.665627 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.665631 | controller |       found'
2026-03-14 09:56:26.665636 | controller |     - '    logger.go:42: 09:39:24 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.665640 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.665644 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.665649 | controller |     - '        '
2026-03-14 09:56:26.665653 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.665658 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.665662 | controller |     - '        '
2026-03-14 09:56:26.665667 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.665672 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.665676 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.665680 | controller |     - '          exit 1'
2026-03-14 09:56:26.665685 | controller |     - '        fi'
2026-03-14 09:56:26.665689 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.665693 | controller |     - '        '
2026-03-14 09:56:26.665699 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.665704 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.665708 | controller |       | base64 -d)'
2026-03-14 09:56:26.665712 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.665717 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.665721 | controller |     - '          exit 1'
2026-03-14 09:56:26.665725 | controller |     - '        fi'
2026-03-14 09:56:26.665730 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.665734 | controller |     - '        '
2026-03-14 09:56:26.665738 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.665743 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.665747 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.665754 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.665758 | controller |       --timeout=300s'
2026-03-14 09:56:26.665766 | controller |     - '        '
2026-03-14 09:56:26.665772 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.665779 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.665785 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.665791 | controller |     - '        '
2026-03-14 09:56:26.665795 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.665800 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.665806 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.665811 | controller |     - '        '
2026-03-14 09:56:26.665815 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.665819 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.665824 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.665828 | controller |       available"'
2026-03-14 09:56:26.665832 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.665836 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.665841 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.665845 | controller |     - '          exit 1'
2026-03-14 09:56:26.665849 | controller |     - '        fi'
2026-03-14 09:56:26.665854 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.665858 | controller |     - '        '
2026-03-14 09:56:26.665862 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.665867 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.665871 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.665875 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.665880 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.665884 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.665890 | controller |       -- \'
2026-03-14 09:56:26.665902 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.665910 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.665915 | controller |     - '    logger.go:42: 09:39:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.665919 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.665924 | controller |     - '    logger.go:42: 09:39:24 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.665928 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.665933 | controller |       found'
2026-03-14 09:56:26.665937 | controller |     - '    logger.go:42: 09:39:25 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.665941 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.665946 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.665950 | controller |     - '        '
2026-03-14 09:56:26.665954 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.665959 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.665963 | controller |     - '        '
2026-03-14 09:56:26.665967 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.665972 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.665976 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.665980 | controller |     - '          exit 1'
2026-03-14 09:56:26.665985 | controller |     - '        fi'
2026-03-14 09:56:26.665989 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.665993 | controller |     - '        '
2026-03-14 09:56:26.665998 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.666002 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.666006 | controller |       | base64 -d)'
2026-03-14 09:56:26.666011 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.666015 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.666019 | controller |     - '          exit 1'
2026-03-14 09:56:26.666024 | controller |     - '        fi'
2026-03-14 09:56:26.666028 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.666032 | controller |     - '        '
2026-03-14 09:56:26.666037 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.666041 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.666045 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.666050 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.666054 | controller |       --timeout=300s'
2026-03-14 09:56:26.666058 | controller |     - '        '
2026-03-14 09:56:26.666063 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.666067 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.666071 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.666076 | controller |     - '        '
2026-03-14 09:56:26.666080 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.666084 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.666089 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.666093 | controller |     - '        '
2026-03-14 09:56:26.666097 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.666103 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.666108 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.666112 | controller |       available"'
2026-03-14 09:56:26.666116 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.666121 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.666128 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.666133 | controller |     - '          exit 1'
2026-03-14 09:56:26.666137 | controller |     - '        fi'
2026-03-14 09:56:26.666142 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.666146 | controller |     - '        '
2026-03-14 09:56:26.666150 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.666155 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.666159 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.666163 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.666168 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.666172 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.666176 | controller |       -- \'
2026-03-14 09:56:26.666187 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.666192 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.666196 | controller |     - '    logger.go:42: 09:39:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.666201 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.666205 | controller |     - '    logger.go:42: 09:39:25 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.666209 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.666214 | controller |       found'
2026-03-14 09:56:26.666218 | controller |     - '    logger.go:42: 09:39:26 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.666222 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.666227 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.666231 | controller |     - '        '
2026-03-14 09:56:26.666235 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.666240 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.666244 | controller |     - '        '
2026-03-14 09:56:26.666248 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.666253 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.666257 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.666261 | controller |     - '          exit 1'
2026-03-14 09:56:26.666266 | controller |     - '        fi'
2026-03-14 09:56:26.666270 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.666274 | controller |     - '        '
2026-03-14 09:56:26.666279 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.666283 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.666287 | controller |       | base64 -d)'
2026-03-14 09:56:26.666292 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.666296 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.666300 | controller |     - '          exit 1'
2026-03-14 09:56:26.666305 | controller |     - '        fi'
2026-03-14 09:56:26.666311 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.666315 | controller |     - '        '
2026-03-14 09:56:26.666320 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.666324 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.666328 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.666333 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.666346 | controller |       --timeout=300s'
2026-03-14 09:56:26.666351 | controller |     - '        '
2026-03-14 09:56:26.666355 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.666359 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.666364 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.666368 | controller |     - '        '
2026-03-14 09:56:26.666372 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.666377 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.666381 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.666385 | controller |     - '        '
2026-03-14 09:56:26.666390 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.666394 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.666398 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.666403 | controller |       available"'
2026-03-14 09:56:26.666407 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.666411 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.666415 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.666420 | controller |     - '          exit 1'
2026-03-14 09:56:26.666424 | controller |     - '        fi'
2026-03-14 09:56:26.666428 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.666432 | controller |     - '        '
2026-03-14 09:56:26.666437 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.666441 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.666445 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.666449 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.666454 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.666458 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.666462 | controller |       -- \'
2026-03-14 09:56:26.666466 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.666477 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.666482 | controller |     - '    logger.go:42: 09:39:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.666486 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.666490 | controller |     - '    logger.go:42: 09:39:26 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.666494 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.666500 | controller |       found'
2026-03-14 09:56:26.666505 | controller |     - '    logger.go:42: 09:39:27 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.666509 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.666513 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.666517 | controller |     - '        '
2026-03-14 09:56:26.666522 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.666526 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.666530 | controller |     - '        '
2026-03-14 09:56:26.666534 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.666539 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.666543 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.666547 | controller |     - '          exit 1'
2026-03-14 09:56:26.666551 | controller |     - '        fi'
2026-03-14 09:56:26.666555 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.666560 | controller |     - '        '
2026-03-14 09:56:26.666564 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.666568 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.666572 | controller |       | base64 -d)'
2026-03-14 09:56:26.666577 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.666581 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.666585 | controller |     - '          exit 1'
2026-03-14 09:56:26.666589 | controller |     - '        fi'
2026-03-14 09:56:26.666607 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.666614 | controller |     - '        '
2026-03-14 09:56:26.666618 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.666623 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.666627 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.666631 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.666635 | controller |       --timeout=300s'
2026-03-14 09:56:26.666640 | controller |     - '        '
2026-03-14 09:56:26.666644 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.666648 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.666653 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.666657 | controller |     - '        '
2026-03-14 09:56:26.666661 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.666665 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.666670 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.666674 | controller |     - '        '
2026-03-14 09:56:26.666678 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.666682 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.666687 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.666691 | controller |       available"'
2026-03-14 09:56:26.666695 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.666699 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.666704 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.666710 | controller |     - '          exit 1'
2026-03-14 09:56:26.666714 | controller |     - '        fi'
2026-03-14 09:56:26.666718 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.666723 | controller |     - '        '
2026-03-14 09:56:26.666727 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.666731 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.666735 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.666740 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.666744 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.666748 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.666753 | controller |       -- \'
2026-03-14 09:56:26.666757 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.666773 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.666780 | controller |     - '    logger.go:42: 09:39:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.666786 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.666792 | controller |     - '    logger.go:42: 09:39:27 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.666796 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.666800 | controller |       found'
2026-03-14 09:56:26.666804 | controller |     - '    logger.go:42: 09:39:28 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.666809 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.666813 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.666817 | controller |     - '        '
2026-03-14 09:56:26.666822 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.666826 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.666830 | controller |     - '        '
2026-03-14 09:56:26.666834 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.666839 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.666843 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.666847 | controller |     - '          exit 1'
2026-03-14 09:56:26.666851 | controller |     - '        fi'
2026-03-14 09:56:26.666856 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.666860 | controller |     - '        '
2026-03-14 09:56:26.666867 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.666871 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.666875 | controller |       | base64 -d)'
2026-03-14 09:56:26.666879 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.666884 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.666888 | controller |     - '          exit 1'
2026-03-14 09:56:26.666892 | controller |     - '        fi'
2026-03-14 09:56:26.666896 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.666901 | controller |     - '        '
2026-03-14 09:56:26.666905 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.666909 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.666913 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.666917 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.666924 | controller |       --timeout=300s'
2026-03-14 09:56:26.666928 | controller |     - '        '
2026-03-14 09:56:26.666932 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.666937 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.666941 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.666945 | controller |     - '        '
2026-03-14 09:56:26.666949 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.666954 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.666958 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.666962 | controller |     - '        '
2026-03-14 09:56:26.666990 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.666998 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.667004 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.667009 | controller |       available"'
2026-03-14 09:56:26.667015 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.667020 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.667024 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.667028 | controller |     - '          exit 1'
2026-03-14 09:56:26.667032 | controller |     - '        fi'
2026-03-14 09:56:26.667037 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.667041 | controller |     - '        '
2026-03-14 09:56:26.667045 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.667049 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.667054 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.667058 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.667062 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.667066 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.667071 | controller |       -- \'
2026-03-14 09:56:26.667075 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.667079 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.667091 | controller |     - '    logger.go:42: 09:39:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.667095 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.667100 | controller |     - '    logger.go:42: 09:39:28 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.667104 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.667108 | controller |       found'
2026-03-14 09:56:26.667113 | controller |     - '    logger.go:42: 09:39:29 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.667117 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.667123 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.667128 | controller |     - '        '
2026-03-14 09:56:26.667132 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.667137 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.667141 | controller |     - '        '
2026-03-14 09:56:26.667147 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.667151 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.667156 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.667160 | controller |     - '          exit 1'
2026-03-14 09:56:26.667164 | controller |     - '        fi'
2026-03-14 09:56:26.667169 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.667173 | controller |     - '        '
2026-03-14 09:56:26.667177 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.667181 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.667186 | controller |       | base64 -d)'
2026-03-14 09:56:26.667190 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.667194 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.667200 | controller |     - '          exit 1'
2026-03-14 09:56:26.667206 | controller |     - '        fi'
2026-03-14 09:56:26.667212 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.667218 | controller |     - '        '
2026-03-14 09:56:26.667223 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.667227 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.667231 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.667235 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.667239 | controller |       --timeout=300s'
2026-03-14 09:56:26.667244 | controller |     - '        '
2026-03-14 09:56:26.667248 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.667254 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.667259 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.667263 | controller |     - '        '
2026-03-14 09:56:26.667267 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.667271 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.667275 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.667280 | controller |     - '        '
2026-03-14 09:56:26.667284 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.667288 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.667292 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.667297 | controller |       available"'
2026-03-14 09:56:26.667301 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.667305 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.667309 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.667313 | controller |     - '          exit 1'
2026-03-14 09:56:26.667318 | controller |     - '        fi'
2026-03-14 09:56:26.667322 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.667326 | controller |     - '        '
2026-03-14 09:56:26.667330 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.667335 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.667339 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.667345 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.667349 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.667353 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.667358 | controller |       -- \'
2026-03-14 09:56:26.667362 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.667366 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.667378 | controller |     - '    logger.go:42: 09:39:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.667382 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.667387 | controller |     - '    logger.go:42: 09:39:30 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.667391 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.667395 | controller |       found'
2026-03-14 09:56:26.667399 | controller |     - '    logger.go:42: 09:39:31 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.667404 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.667408 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.667412 | controller |     - '        '
2026-03-14 09:56:26.667416 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.667421 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.667425 | controller |     - '        '
2026-03-14 09:56:26.667429 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.667433 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.667438 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.667442 | controller |     - '          exit 1'
2026-03-14 09:56:26.667446 | controller |     - '        fi'
2026-03-14 09:56:26.667450 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.667455 | controller |     - '        '
2026-03-14 09:56:26.667459 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.667465 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.667469 | controller |       | base64 -d)'
2026-03-14 09:56:26.667473 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.667478 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.667482 | controller |     - '          exit 1'
2026-03-14 09:56:26.667486 | controller |     - '        fi'
2026-03-14 09:56:26.667490 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.667495 | controller |     - '        '
2026-03-14 09:56:26.667499 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.667503 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.667507 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.667512 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.667516 | controller |       --timeout=300s'
2026-03-14 09:56:26.667520 | controller |     - '        '
2026-03-14 09:56:26.667524 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.667529 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.667533 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.667537 | controller |     - '        '
2026-03-14 09:56:26.667541 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.667547 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.667552 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.667556 | controller |     - '        '
2026-03-14 09:56:26.667560 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.667564 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.667569 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.667577 | controller |       available"'
2026-03-14 09:56:26.667581 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.667585 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.667589 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.667594 | controller |     - '          exit 1'
2026-03-14 09:56:26.667612 | controller |     - '        fi'
2026-03-14 09:56:26.667617 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.667621 | controller |     - '        '
2026-03-14 09:56:26.667625 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.667629 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.667634 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.667638 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.667642 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.667646 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.667651 | controller |       -- \'
2026-03-14 09:56:26.667655 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.667659 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.667663 | controller |     - '    logger.go:42: 09:39:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.667674 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.667679 | controller |     - '    logger.go:42: 09:39:31 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.667683 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.667687 | controller |       found'
2026-03-14 09:56:26.667692 | controller |     - '    logger.go:42: 09:39:32 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.667696 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.667700 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.667704 | controller |     - '        '
2026-03-14 09:56:26.667709 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.667713 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.667717 | controller |     - '        '
2026-03-14 09:56:26.667721 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.667726 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.667730 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.667734 | controller |     - '          exit 1'
2026-03-14 09:56:26.667738 | controller |     - '        fi'
2026-03-14 09:56:26.667744 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.667750 | controller |     - '        '
2026-03-14 09:56:26.667756 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.667764 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.667770 | controller |       | base64 -d)'
2026-03-14 09:56:26.667776 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.667783 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.667789 | controller |     - '          exit 1'
2026-03-14 09:56:26.667795 | controller |     - '        fi'
2026-03-14 09:56:26.667801 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.667806 | controller |     - '        '
2026-03-14 09:56:26.667812 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.667817 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.667823 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.667828 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.667834 | controller |       --timeout=300s'
2026-03-14 09:56:26.667881 | controller |     - '        '
2026-03-14 09:56:26.667885 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.667890 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.667894 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.667898 | controller |     - '        '
2026-03-14 09:56:26.667902 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.667907 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.667911 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.667918 | controller |     - '        '
2026-03-14 09:56:26.667922 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.667927 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.667931 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.667935 | controller |       available"'
2026-03-14 09:56:26.667939 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.667944 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.667948 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.667952 | controller |     - '          exit 1'
2026-03-14 09:56:26.667956 | controller |     - '        fi'
2026-03-14 09:56:26.667961 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.667965 | controller |     - '        '
2026-03-14 09:56:26.667971 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.667975 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.667979 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.667983 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.667988 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.667992 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.667996 | controller |       -- \'
2026-03-14 09:56:26.668000 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.668008 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.668012 | controller |     - '    logger.go:42: 09:39:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.668016 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.668030 | controller |     - '    logger.go:42: 09:39:32 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.668036 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.668041 | controller |       found'
2026-03-14 09:56:26.668045 | controller |     - '    logger.go:42: 09:39:33 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.668049 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.668054 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.668058 | controller |     - '        '
2026-03-14 09:56:26.668062 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.668066 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.668071 | controller |     - '        '
2026-03-14 09:56:26.668075 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.668079 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.668083 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.668088 | controller |     - '          exit 1'
2026-03-14 09:56:26.668092 | controller |     - '        fi'
2026-03-14 09:56:26.668096 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.668100 | controller |     - '        '
2026-03-14 09:56:26.668104 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.668109 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.668113 | controller |       | base64 -d)'
2026-03-14 09:56:26.668117 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.668121 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.668125 | controller |     - '          exit 1'
2026-03-14 09:56:26.668130 | controller |     - '        fi'
2026-03-14 09:56:26.668134 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.668138 | controller |     - '        '
2026-03-14 09:56:26.668142 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.668147 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.668151 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.668155 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.668159 | controller |       --timeout=300s'
2026-03-14 09:56:26.668163 | controller |     - '        '
2026-03-14 09:56:26.668168 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.668172 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.668176 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.668180 | controller |     - '        '
2026-03-14 09:56:26.668185 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.668189 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.668193 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.668197 | controller |     - '        '
2026-03-14 09:56:26.668202 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.668206 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.668212 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.668216 | controller |       available"'
2026-03-14 09:56:26.668220 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.668225 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.668229 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.668233 | controller |     - '          exit 1'
2026-03-14 09:56:26.668237 | controller |     - '        fi'
2026-03-14 09:56:26.668242 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.668247 | controller |     - '        '
2026-03-14 09:56:26.668252 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.668256 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.668260 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.668264 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.668269 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.668273 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.668277 | controller |       -- \'
2026-03-14 09:56:26.668281 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.668286 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.668290 | controller |     - '    logger.go:42: 09:39:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.668294 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.668298 | controller |     - '    logger.go:42: 09:39:33 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.668309 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.668313 | controller |       found'
2026-03-14 09:56:26.668318 | controller |     - '    logger.go:42: 09:39:34 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.668322 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.668326 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.668331 | controller |     - '        '
2026-03-14 09:56:26.668335 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.668339 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.668343 | controller |     - '        '
2026-03-14 09:56:26.668347 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.668352 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.668356 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.668360 | controller |     - '          exit 1'
2026-03-14 09:56:26.668364 | controller |     - '        fi'
2026-03-14 09:56:26.668369 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.668373 | controller |     - '        '
2026-03-14 09:56:26.668377 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.668381 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.668385 | controller |       | base64 -d)'
2026-03-14 09:56:26.668390 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.668394 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.668398 | controller |     - '          exit 1'
2026-03-14 09:56:26.668402 | controller |     - '        fi'
2026-03-14 09:56:26.668407 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.668413 | controller |     - '        '
2026-03-14 09:56:26.668417 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.668421 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.668426 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.668430 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.668434 | controller |       --timeout=300s'
2026-03-14 09:56:26.668438 | controller |     - '        '
2026-03-14 09:56:26.668443 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.668447 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.668451 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.668455 | controller |     - '        '
2026-03-14 09:56:26.668460 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.668464 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.668468 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.668472 | controller |     - '        '
2026-03-14 09:56:26.668477 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.668481 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.668485 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.668489 | controller |       available"'
2026-03-14 09:56:26.668494 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.668498 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.668502 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.668506 | controller |     - '          exit 1'
2026-03-14 09:56:26.668511 | controller |     - '        fi'
2026-03-14 09:56:26.668515 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.668519 | controller |     - '        '
2026-03-14 09:56:26.668523 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.668528 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.668532 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.668536 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.668540 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.668545 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.668549 | controller |       -- \'
2026-03-14 09:56:26.668553 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.668557 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.668562 | controller |     - '    logger.go:42: 09:39:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.668566 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.668570 | controller |     - '    logger.go:42: 09:39:34 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.668580 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.668586 | controller |       found'
2026-03-14 09:56:26.668610 | controller |     - '    logger.go:42: 09:39:35 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.668629 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.668634 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.668638 | controller |     - '        '
2026-03-14 09:56:26.668642 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.668646 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.668651 | controller |     - '        '
2026-03-14 09:56:26.668655 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.668659 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.668664 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.668670 | controller |     - '          exit 1'
2026-03-14 09:56:26.668677 | controller |     - '        fi'
2026-03-14 09:56:26.668683 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.668692 | controller |     - '        '
2026-03-14 09:56:26.668696 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.668700 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.668706 | controller |       | base64 -d)'
2026-03-14 09:56:26.668710 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.668715 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.668719 | controller |     - '          exit 1'
2026-03-14 09:56:26.668723 | controller |     - '        fi'
2026-03-14 09:56:26.668727 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.668732 | controller |     - '        '
2026-03-14 09:56:26.668736 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.668740 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.668744 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.668749 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.668753 | controller |       --timeout=300s'
2026-03-14 09:56:26.668757 | controller |     - '        '
2026-03-14 09:56:26.668762 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.668768 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.668774 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.668780 | controller |     - '        '
2026-03-14 09:56:26.668786 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.668792 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.668797 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.668802 | controller |     - '        '
2026-03-14 09:56:26.668806 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.668810 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.668814 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.668819 | controller |       available"'
2026-03-14 09:56:26.668823 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.668827 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.668834 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.668838 | controller |     - '          exit 1'
2026-03-14 09:56:26.668842 | controller |     - '        fi'
2026-03-14 09:56:26.668849 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.668853 | controller |     - '        '
2026-03-14 09:56:26.668857 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.668863 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.668868 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.668872 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.668876 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.668880 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.668884 | controller |       -- \'
2026-03-14 09:56:26.668889 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.668893 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.668897 | controller |     - '    logger.go:42: 09:39:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.668902 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.668906 | controller |     - '    logger.go:42: 09:39:35 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.668910 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.668914 | controller |       found'
2026-03-14 09:56:26.668929 | controller |     - '    logger.go:42: 09:39:36 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.668934 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.668938 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.668942 | controller |     - '        '
2026-03-14 09:56:26.668947 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.668951 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.668955 | controller |     - '        '
2026-03-14 09:56:26.668959 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.668963 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.668968 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.668972 | controller |     - '          exit 1'
2026-03-14 09:56:26.668976 | controller |     - '        fi'
2026-03-14 09:56:26.668980 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.668985 | controller |     - '        '
2026-03-14 09:56:26.668989 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.668993 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.668997 | controller |       | base64 -d)'
2026-03-14 09:56:26.669002 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.669006 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.669010 | controller |     - '          exit 1'
2026-03-14 09:56:26.669014 | controller |     - '        fi'
2026-03-14 09:56:26.669019 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.669024 | controller |     - '        '
2026-03-14 09:56:26.669028 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.669032 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.669036 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.669041 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.669045 | controller |       --timeout=300s'
2026-03-14 09:56:26.669049 | controller |     - '        '
2026-03-14 09:56:26.669057 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.669061 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.669069 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.669073 | controller |     - '        '
2026-03-14 09:56:26.669078 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.669082 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.669086 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.669090 | controller |     - '        '
2026-03-14 09:56:26.669095 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.669099 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.669103 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.669107 | controller |       available"'
2026-03-14 09:56:26.669112 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.669116 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.669120 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.669124 | controller |     - '          exit 1'
2026-03-14 09:56:26.669128 | controller |     - '        fi'
2026-03-14 09:56:26.669133 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.669137 | controller |     - '        '
2026-03-14 09:56:26.669141 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.669145 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.669150 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.669154 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.669158 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.669162 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.669167 | controller |       -- \'
2026-03-14 09:56:26.669171 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.669176 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.669182 | controller |     - '    logger.go:42: 09:39:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.669188 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.669194 | controller |     - '    logger.go:42: 09:39:37 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.669200 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.669206 | controller |       found'
2026-03-14 09:56:26.669218 | controller |     - '    logger.go:42: 09:39:38 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.669222 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.669226 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.669231 | controller |     - '        '
2026-03-14 09:56:26.669235 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.669239 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.669244 | controller |     - '        '
2026-03-14 09:56:26.669248 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.669254 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.669259 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.669263 | controller |     - '          exit 1'
2026-03-14 09:56:26.669267 | controller |     - '        fi'
2026-03-14 09:56:26.669271 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.669276 | controller |     - '        '
2026-03-14 09:56:26.669280 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.669284 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.669288 | controller |       | base64 -d)'
2026-03-14 09:56:26.669293 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.669297 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.669301 | controller |     - '          exit 1'
2026-03-14 09:56:26.669305 | controller |     - '        fi'
2026-03-14 09:56:26.669309 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.669314 | controller |     - '        '
2026-03-14 09:56:26.669318 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.669322 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.669326 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.669331 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.669336 | controller |       --timeout=300s'
2026-03-14 09:56:26.669341 | controller |     - '        '
2026-03-14 09:56:26.669346 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.669350 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.669355 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.669361 | controller |     - '        '
2026-03-14 09:56:26.669365 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.669370 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.669374 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.669378 | controller |     - '        '
2026-03-14 09:56:26.669382 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.669386 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.669391 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.669395 | controller |       available"'
2026-03-14 09:56:26.669399 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.669403 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.669407 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.669412 | controller |     - '          exit 1'
2026-03-14 09:56:26.669416 | controller |     - '        fi'
2026-03-14 09:56:26.669420 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.669424 | controller |     - '        '
2026-03-14 09:56:26.669429 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.669433 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.669437 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.669441 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.669449 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.669454 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.669458 | controller |       -- \'
2026-03-14 09:56:26.669462 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.669466 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.669471 | controller |     - '    logger.go:42: 09:39:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.669475 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.669479 | controller |     - '    logger.go:42: 09:39:38 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.669483 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.669488 | controller |       found'
2026-03-14 09:56:26.669492 | controller |     - '    logger.go:42: 09:39:39 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.669496 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.669507 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.669511 | controller |     - '        '
2026-03-14 09:56:26.669515 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.669520 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.669524 | controller |     - '        '
2026-03-14 09:56:26.669528 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.669533 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.669537 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.669541 | controller |     - '          exit 1'
2026-03-14 09:56:26.669545 | controller |     - '        fi'
2026-03-14 09:56:26.669550 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.669554 | controller |     - '        '
2026-03-14 09:56:26.669558 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.669562 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.669567 | controller |       | base64 -d)'
2026-03-14 09:56:26.669571 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.669575 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.669579 | controller |     - '          exit 1'
2026-03-14 09:56:26.669584 | controller |     - '        fi'
2026-03-14 09:56:26.669588 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.669592 | controller |     - '        '
2026-03-14 09:56:26.669611 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.669615 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.669619 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.669624 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.669628 | controller |       --timeout=300s'
2026-03-14 09:56:26.669632 | controller |     - '        '
2026-03-14 09:56:26.669636 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.669641 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.669646 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.669652 | controller |     - '        '
2026-03-14 09:56:26.669657 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.669665 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.669670 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.669675 | controller |     - '        '
2026-03-14 09:56:26.669681 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.669687 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.669692 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.669698 | controller |       available"'
2026-03-14 09:56:26.669703 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.669709 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.669714 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.669720 | controller |     - '          exit 1'
2026-03-14 09:56:26.669725 | controller |     - '        fi'
2026-03-14 09:56:26.669731 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.669736 | controller |     - '        '
2026-03-14 09:56:26.669742 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.669747 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.669753 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.669758 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.669764 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.669770 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.669775 | controller |       -- \'
2026-03-14 09:56:26.669781 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.669786 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.669793 | controller |     - '    logger.go:42: 09:39:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.669798 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.669804 | controller |     - '    logger.go:42: 09:39:39 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.669809 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.669815 | controller |       found'
2026-03-14 09:56:26.669821 | controller |     - '    logger.go:42: 09:39:40 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.669826 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.669832 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.669838 | controller |     - '        '
2026-03-14 09:56:26.669854 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.669861 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.669866 | controller |     - '        '
2026-03-14 09:56:26.669872 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.669878 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.669884 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.669890 | controller |     - '          exit 1'
2026-03-14 09:56:26.669896 | controller |     - '        fi'
2026-03-14 09:56:26.669902 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.669907 | controller |     - '        '
2026-03-14 09:56:26.669913 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.669919 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.669928 | controller |       | base64 -d)'
2026-03-14 09:56:26.669934 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.669940 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.669946 | controller |     - '          exit 1'
2026-03-14 09:56:26.669952 | controller |     - '        fi'
2026-03-14 09:56:26.669957 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.669963 | controller |     - '        '
2026-03-14 09:56:26.669969 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.669975 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.669980 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.669986 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.669992 | controller |       --timeout=300s'
2026-03-14 09:56:26.669998 | controller |     - '        '
2026-03-14 09:56:26.670004 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.670009 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.670015 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.670021 | controller |     - '        '
2026-03-14 09:56:26.670027 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.670033 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.670039 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.670044 | controller |     - '        '
2026-03-14 09:56:26.670050 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.670055 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.670061 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.670066 | controller |       available"'
2026-03-14 09:56:26.670072 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.670078 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.670084 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.670089 | controller |     - '          exit 1'
2026-03-14 09:56:26.670095 | controller |     - '        fi'
2026-03-14 09:56:26.670101 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.670107 | controller |     - '        '
2026-03-14 09:56:26.670113 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.670119 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.670126 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.670137 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.670143 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.670149 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.670157 | controller |       -- \'
2026-03-14 09:56:26.670163 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.670169 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.670180 | controller |     - '    logger.go:42: 09:39:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.670185 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.670191 | controller |     - '    logger.go:42: 09:39:40 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.670197 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.670203 | controller |       found'
2026-03-14 09:56:26.670208 | controller |     - '    logger.go:42: 09:39:41 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.670214 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.670220 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.670225 | controller |     - '        '
2026-03-14 09:56:26.670241 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.670245 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.670250 | controller |     - '        '
2026-03-14 09:56:26.670254 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.670258 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.670262 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.670267 | controller |     - '          exit 1'
2026-03-14 09:56:26.670271 | controller |     - '        fi'
2026-03-14 09:56:26.670275 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.670279 | controller |     - '        '
2026-03-14 09:56:26.670284 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.670290 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.670295 | controller |       | base64 -d)'
2026-03-14 09:56:26.670299 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.670303 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.670307 | controller |     - '          exit 1'
2026-03-14 09:56:26.670313 | controller |     - '        fi'
2026-03-14 09:56:26.670318 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.670322 | controller |     - '        '
2026-03-14 09:56:26.670326 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.670330 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.670335 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.670339 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.670343 | controller |       --timeout=300s'
2026-03-14 09:56:26.670347 | controller |     - '        '
2026-03-14 09:56:26.670351 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.670356 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.670360 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.670364 | controller |     - '        '
2026-03-14 09:56:26.670368 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.670373 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.670377 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.670381 | controller |     - '        '
2026-03-14 09:56:26.670385 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.670390 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.670394 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.670400 | controller |       available"'
2026-03-14 09:56:26.670405 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.670409 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.670413 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.670417 | controller |     - '          exit 1'
2026-03-14 09:56:26.670422 | controller |     - '        fi'
2026-03-14 09:56:26.670426 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.670430 | controller |     - '        '
2026-03-14 09:56:26.670434 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.670439 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.670443 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.670447 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.670451 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.670456 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.670460 | controller |       -- \'
2026-03-14 09:56:26.670464 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.670468 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.670473 | controller |     - '    logger.go:42: 09:39:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.670477 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.670481 | controller |     - '    logger.go:42: 09:39:41 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.670485 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.670490 | controller |       found'
2026-03-14 09:56:26.670494 | controller |     - '    logger.go:42: 09:39:42 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.670498 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.670502 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.670507 | controller |     - '        '
2026-03-14 09:56:26.670511 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.670521 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.670526 | controller |     - '        '
2026-03-14 09:56:26.670530 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.670534 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.670539 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.670543 | controller |     - '          exit 1'
2026-03-14 09:56:26.670547 | controller |     - '        fi'
2026-03-14 09:56:26.670552 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.670556 | controller |     - '        '
2026-03-14 09:56:26.670563 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.670568 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.670572 | controller |       | base64 -d)'
2026-03-14 09:56:26.670576 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.670581 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.670585 | controller |     - '          exit 1'
2026-03-14 09:56:26.670589 | controller |     - '        fi'
2026-03-14 09:56:26.670605 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.670622 | controller |     - '        '
2026-03-14 09:56:26.670629 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.670633 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.670637 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.670642 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.670646 | controller |       --timeout=300s'
2026-03-14 09:56:26.670650 | controller |     - '        '
2026-03-14 09:56:26.670654 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.670659 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.670663 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.670667 | controller |     - '        '
2026-03-14 09:56:26.670671 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.670676 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.670680 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.670684 | controller |     - '        '
2026-03-14 09:56:26.670688 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.670693 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.670697 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.670701 | controller |       available"'
2026-03-14 09:56:26.670705 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.670710 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.670714 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.670718 | controller |     - '          exit 1'
2026-03-14 09:56:26.670722 | controller |     - '        fi'
2026-03-14 09:56:26.670726 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.670731 | controller |     - '        '
2026-03-14 09:56:26.670735 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.670739 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.670743 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.670748 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.670752 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.670756 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.670760 | controller |       -- \'
2026-03-14 09:56:26.670765 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.670769 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.670773 | controller |     - '    logger.go:42: 09:39:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.670777 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.670781 | controller |     - '    logger.go:42: 09:39:43 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.670788 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.670792 | controller |       found'
2026-03-14 09:56:26.670796 | controller |     - '    logger.go:42: 09:39:44 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.670802 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.670806 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.670810 | controller |     - '        '
2026-03-14 09:56:26.670815 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.670827 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.670832 | controller |     - '        '
2026-03-14 09:56:26.670836 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.670841 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.670847 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.670852 | controller |     - '          exit 1'
2026-03-14 09:56:26.670856 | controller |     - '        fi'
2026-03-14 09:56:26.670860 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.670865 | controller |     - '        '
2026-03-14 09:56:26.670869 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.670874 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.670879 | controller |       | base64 -d)'
2026-03-14 09:56:26.670885 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.670891 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.670895 | controller |     - '          exit 1'
2026-03-14 09:56:26.670899 | controller |     - '        fi'
2026-03-14 09:56:26.670903 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.670908 | controller |     - '        '
2026-03-14 09:56:26.670912 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.670916 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.670921 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.670925 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.670929 | controller |       --timeout=300s'
2026-03-14 09:56:26.670933 | controller |     - '        '
2026-03-14 09:56:26.670938 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.670942 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.670946 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.670950 | controller |     - '        '
2026-03-14 09:56:26.670955 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.670959 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.670963 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.670992 | controller |     - '        '
2026-03-14 09:56:26.670996 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.671001 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.671005 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.671009 | controller |       available"'
2026-03-14 09:56:26.671013 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.671018 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.671022 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.671026 | controller |     - '          exit 1'
2026-03-14 09:56:26.671030 | controller |     - '        fi'
2026-03-14 09:56:26.671035 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.671041 | controller |     - '        '
2026-03-14 09:56:26.671046 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.671050 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.671054 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.671058 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.671063 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.671067 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.671071 | controller |       -- \'
2026-03-14 09:56:26.671075 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.671080 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.671084 | controller |     - '    logger.go:42: 09:39:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.671088 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.671093 | controller |     - '    logger.go:42: 09:39:44 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.671097 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.671102 | controller |       found'
2026-03-14 09:56:26.671106 | controller |     - '    logger.go:42: 09:39:45 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.671110 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.671115 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.671119 | controller |     - '        '
2026-03-14 09:56:26.671123 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.671127 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.671132 | controller |     - '        '
2026-03-14 09:56:26.671143 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.671148 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.671152 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.671156 | controller |     - '          exit 1'
2026-03-14 09:56:26.671161 | controller |     - '        fi'
2026-03-14 09:56:26.671165 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.671169 | controller |     - '        '
2026-03-14 09:56:26.671174 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.671178 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.671182 | controller |       | base64 -d)'
2026-03-14 09:56:26.671186 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.671190 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.671195 | controller |     - '          exit 1'
2026-03-14 09:56:26.671199 | controller |     - '        fi'
2026-03-14 09:56:26.671203 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.671207 | controller |     - '        '
2026-03-14 09:56:26.671212 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.671216 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.671220 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.671225 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.671229 | controller |       --timeout=300s'
2026-03-14 09:56:26.671233 | controller |     - '        '
2026-03-14 09:56:26.671237 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.671244 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.671248 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.671255 | controller |     - '        '
2026-03-14 09:56:26.671259 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.671263 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.671268 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.671272 | controller |     - '        '
2026-03-14 09:56:26.671276 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.671280 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.671285 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.671289 | controller |       available"'
2026-03-14 09:56:26.671293 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.671297 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.671302 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.671306 | controller |     - '          exit 1'
2026-03-14 09:56:26.671310 | controller |     - '        fi'
2026-03-14 09:56:26.671314 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.671319 | controller |     - '        '
2026-03-14 09:56:26.671323 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.671327 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.671331 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.671335 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.671340 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.671344 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.671348 | controller |       -- \'
2026-03-14 09:56:26.671353 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.671357 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.671361 | controller |     - '    logger.go:42: 09:39:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.671365 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.671369 | controller |     - '    logger.go:42: 09:39:45 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.671374 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.671380 | controller |       found'
2026-03-14 09:56:26.671384 | controller |     - '    logger.go:42: 09:39:46 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.671388 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.671392 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.671397 | controller |     - '        '
2026-03-14 09:56:26.671401 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.671405 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.671410 | controller |     - '        '
2026-03-14 09:56:26.671422 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.671428 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.671432 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.671437 | controller |     - '          exit 1'
2026-03-14 09:56:26.671441 | controller |     - '        fi'
2026-03-14 09:56:26.671445 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.671450 | controller |     - '        '
2026-03-14 09:56:26.671454 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.671458 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.671462 | controller |       | base64 -d)'
2026-03-14 09:56:26.671467 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.671471 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.671475 | controller |     - '          exit 1'
2026-03-14 09:56:26.671479 | controller |     - '        fi'
2026-03-14 09:56:26.671483 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.671488 | controller |     - '        '
2026-03-14 09:56:26.671492 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.671496 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.671500 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.671505 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.671509 | controller |       --timeout=300s'
2026-03-14 09:56:26.671513 | controller |     - '        '
2026-03-14 09:56:26.671517 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.671523 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.671527 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.671532 | controller |     - '        '
2026-03-14 09:56:26.671536 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.671540 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.671544 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.671549 | controller |     - '        '
2026-03-14 09:56:26.671553 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.671557 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.671561 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.671565 | controller |       available"'
2026-03-14 09:56:26.671570 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.671574 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.671580 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.671584 | controller |     - '          exit 1'
2026-03-14 09:56:26.671588 | controller |     - '        fi'
2026-03-14 09:56:26.671593 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.671611 | controller |     - '        '
2026-03-14 09:56:26.671615 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.671619 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.671623 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.671628 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.671634 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.671638 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.671642 | controller |       -- \'
2026-03-14 09:56:26.671647 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.671651 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.671655 | controller |     - '    logger.go:42: 09:39:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.671659 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.671663 | controller |     - '    logger.go:42: 09:39:46 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.671668 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.671672 | controller |       found'
2026-03-14 09:56:26.671676 | controller |     - '    logger.go:42: 09:39:47 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.671680 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.671685 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.671689 | controller |     - '        '
2026-03-14 09:56:26.671693 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.671697 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.671701 | controller |     - '        '
2026-03-14 09:56:26.671706 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.671716 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.671720 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.671725 | controller |     - '          exit 1'
2026-03-14 09:56:26.671729 | controller |     - '        fi'
2026-03-14 09:56:26.671733 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.671737 | controller |     - '        '
2026-03-14 09:56:26.671742 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.671746 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.671750 | controller |       | base64 -d)'
2026-03-14 09:56:26.671754 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.671758 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.671763 | controller |     - '          exit 1'
2026-03-14 09:56:26.671767 | controller |     - '        fi'
2026-03-14 09:56:26.671771 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.671776 | controller |     - '        '
2026-03-14 09:56:26.671780 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.671784 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.671788 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.671792 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.671797 | controller |       --timeout=300s'
2026-03-14 09:56:26.671801 | controller |     - '        '
2026-03-14 09:56:26.671805 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.671819 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.671824 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.671828 | controller |     - '        '
2026-03-14 09:56:26.671832 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.671836 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.671851 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.671857 | controller |     - '        '
2026-03-14 09:56:26.671863 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.671867 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.671871 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.671875 | controller |       available"'
2026-03-14 09:56:26.671880 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.671884 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.671888 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.671892 | controller |     - '          exit 1'
2026-03-14 09:56:26.671896 | controller |     - '        fi'
2026-03-14 09:56:26.671901 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.671905 | controller |     - '        '
2026-03-14 09:56:26.671909 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.671913 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.671917 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.671922 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.671926 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.671930 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.671934 | controller |       -- \'
2026-03-14 09:56:26.671938 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.671943 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.671947 | controller |     - '    logger.go:42: 09:39:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.671951 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.671955 | controller |     - '    logger.go:42: 09:39:47 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.671959 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.671964 | controller |       found'
2026-03-14 09:56:26.671968 | controller |     - '    logger.go:42: 09:39:48 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.671976 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.671980 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.671985 | controller |     - '        '
2026-03-14 09:56:26.671989 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.671993 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.671997 | controller |     - '        '
2026-03-14 09:56:26.672002 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.672006 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.672018 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.672022 | controller |     - '          exit 1'
2026-03-14 09:56:26.672027 | controller |     - '        fi'
2026-03-14 09:56:26.672031 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.672035 | controller |     - '        '
2026-03-14 09:56:26.672039 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.672044 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.672050 | controller |       | base64 -d)'
2026-03-14 09:56:26.672054 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.672058 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.672063 | controller |     - '          exit 1'
2026-03-14 09:56:26.672067 | controller |     - '        fi'
2026-03-14 09:56:26.672071 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.672075 | controller |     - '        '
2026-03-14 09:56:26.672079 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.672084 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.672088 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.672092 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.672096 | controller |       --timeout=300s'
2026-03-14 09:56:26.672101 | controller |     - '        '
2026-03-14 09:56:26.672105 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.672109 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.672113 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.672117 | controller |     - '        '
2026-03-14 09:56:26.672122 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.672126 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.672130 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.672134 | controller |     - '        '
2026-03-14 09:56:26.672139 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.672143 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.672147 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.672151 | controller |       available"'
2026-03-14 09:56:26.672156 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.672160 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.672164 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.672168 | controller |     - '          exit 1'
2026-03-14 09:56:26.672173 | controller |     - '        fi'
2026-03-14 09:56:26.672177 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.672181 | controller |     - '        '
2026-03-14 09:56:26.672185 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.672190 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.672194 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.672198 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.672202 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.672210 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.672214 | controller |       -- \'
2026-03-14 09:56:26.672220 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.672224 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.672228 | controller |     - '    logger.go:42: 09:39:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.672234 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.672238 | controller |     - '    logger.go:42: 09:39:48 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.672243 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.672247 | controller |       found'
2026-03-14 09:56:26.672251 | controller |     - '    logger.go:42: 09:39:49 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.672255 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.672260 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.672264 | controller |     - '        '
2026-03-14 09:56:26.672268 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.672272 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.672276 | controller |     - '        '
2026-03-14 09:56:26.672281 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.672285 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.672289 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.672293 | controller |     - '          exit 1'
2026-03-14 09:56:26.672304 | controller |     - '        fi'
2026-03-14 09:56:26.672308 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.672313 | controller |     - '        '
2026-03-14 09:56:26.672317 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.672321 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.672325 | controller |       | base64 -d)'
2026-03-14 09:56:26.672330 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.672334 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.672338 | controller |     - '          exit 1'
2026-03-14 09:56:26.672342 | controller |     - '        fi'
2026-03-14 09:56:26.672347 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.672351 | controller |     - '        '
2026-03-14 09:56:26.672357 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.672362 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.672366 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.672371 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.672376 | controller |       --timeout=300s'
2026-03-14 09:56:26.672381 | controller |     - '        '
2026-03-14 09:56:26.672386 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.672390 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.672394 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.672398 | controller |     - '        '
2026-03-14 09:56:26.672403 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.672407 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.672411 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.672416 | controller |     - '        '
2026-03-14 09:56:26.672420 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.672424 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.672428 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.672435 | controller |       available"'
2026-03-14 09:56:26.672448 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.672453 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.672457 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.672461 | controller |     - '          exit 1'
2026-03-14 09:56:26.672465 | controller |     - '        fi'
2026-03-14 09:56:26.672470 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.672474 | controller |     - '        '
2026-03-14 09:56:26.672478 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.672482 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.672487 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.672491 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.672495 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.672499 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.672503 | controller |       -- \'
2026-03-14 09:56:26.672508 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.672512 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.672516 | controller |     - '    logger.go:42: 09:39:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.672520 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.672525 | controller |     - '    logger.go:42: 09:39:49 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.672529 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.672533 | controller |       found'
2026-03-14 09:56:26.672537 | controller |     - '    logger.go:42: 09:39:50 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.672542 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.672546 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.672556 | controller |     - '        '
2026-03-14 09:56:26.672560 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.672567 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.672571 | controller |     - '        '
2026-03-14 09:56:26.672576 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.672584 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.672589 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.672608 | controller |     - '          exit 1'
2026-03-14 09:56:26.672615 | controller |     - '        fi'
2026-03-14 09:56:26.672625 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.672629 | controller |     - '        '
2026-03-14 09:56:26.672633 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.672638 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.672642 | controller |       | base64 -d)'
2026-03-14 09:56:26.672646 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.672650 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.672655 | controller |     - '          exit 1'
2026-03-14 09:56:26.672659 | controller |     - '        fi'
2026-03-14 09:56:26.672663 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.672667 | controller |     - '        '
2026-03-14 09:56:26.672672 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.672678 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.672682 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.672686 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.672691 | controller |       --timeout=300s'
2026-03-14 09:56:26.672695 | controller |     - '        '
2026-03-14 09:56:26.672699 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.672703 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.672708 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.672712 | controller |     - '        '
2026-03-14 09:56:26.672716 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.672720 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.672725 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.672729 | controller |     - '        '
2026-03-14 09:56:26.672733 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.672737 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.672741 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.672746 | controller |       available"'
2026-03-14 09:56:26.672750 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.672754 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.672758 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.672762 | controller |     - '          exit 1'
2026-03-14 09:56:26.672767 | controller |     - '        fi'
2026-03-14 09:56:26.672771 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.672775 | controller |     - '        '
2026-03-14 09:56:26.672779 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.672784 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.672788 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.672792 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.672796 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.672800 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.672805 | controller |       -- \'
2026-03-14 09:56:26.672809 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.672813 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.672817 | controller |     - '    logger.go:42: 09:39:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.672822 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.672826 | controller |     - '    logger.go:42: 09:39:51 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.672830 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.672834 | controller |       found'
2026-03-14 09:56:26.672839 | controller |     - '    logger.go:42: 09:39:52 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.672843 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.672849 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.672853 | controller |     - '        '
2026-03-14 09:56:26.672858 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.672862 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.672866 | controller |     - '        '
2026-03-14 09:56:26.672870 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.672875 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.672879 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.672883 | controller |     - '          exit 1'
2026-03-14 09:56:26.672887 | controller |     - '        fi'
2026-03-14 09:56:26.672892 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.672898 | controller |     - '        '
2026-03-14 09:56:26.672908 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.672915 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.672920 | controller |       | base64 -d)'
2026-03-14 09:56:26.672925 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.672929 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.672933 | controller |     - '          exit 1'
2026-03-14 09:56:26.672938 | controller |     - '        fi'
2026-03-14 09:56:26.672942 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.672946 | controller |     - '        '
2026-03-14 09:56:26.672951 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.672955 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.672959 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.672963 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.672968 | controller |       --timeout=300s'
2026-03-14 09:56:26.672972 | controller |     - '        '
2026-03-14 09:56:26.672976 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.672982 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.672986 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.672991 | controller |     - '        '
2026-03-14 09:56:26.672995 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.672999 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.673003 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.673008 | controller |     - '        '
2026-03-14 09:56:26.673012 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.673016 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.673020 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.673025 | controller |       available"'
2026-03-14 09:56:26.673029 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.673033 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.673037 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.673042 | controller |     - '          exit 1'
2026-03-14 09:56:26.673046 | controller |     - '        fi'
2026-03-14 09:56:26.673050 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.673056 | controller |     - '        '
2026-03-14 09:56:26.673060 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.673065 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.673069 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.673073 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.673077 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.673082 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.673086 | controller |       -- \'
2026-03-14 09:56:26.673090 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.673094 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.673099 | controller |     - '    logger.go:42: 09:39:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.673103 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.673107 | controller |     - '    logger.go:42: 09:39:52 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.673111 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.673116 | controller |       found'
2026-03-14 09:56:26.673120 | controller |     - '    logger.go:42: 09:39:53 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.673130 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.673135 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.673139 | controller |     - '        '
2026-03-14 09:56:26.673143 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.673147 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.673153 | controller |     - '        '
2026-03-14 09:56:26.673158 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.673162 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.673166 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.673170 | controller |     - '          exit 1'
2026-03-14 09:56:26.673175 | controller |     - '        fi'
2026-03-14 09:56:26.673179 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.673183 | controller |     - '        '
2026-03-14 09:56:26.673199 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.673204 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.673208 | controller |       | base64 -d)'
2026-03-14 09:56:26.673212 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.673217 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.673221 | controller |     - '          exit 1'
2026-03-14 09:56:26.673225 | controller |     - '        fi'
2026-03-14 09:56:26.673229 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.673234 | controller |     - '        '
2026-03-14 09:56:26.673238 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.673242 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.673246 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.673250 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.673255 | controller |       --timeout=300s'
2026-03-14 09:56:26.673259 | controller |     - '        '
2026-03-14 09:56:26.673263 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.673270 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.673275 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.673279 | controller |     - '        '
2026-03-14 09:56:26.673283 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.673288 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.673295 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.673299 | controller |     - '        '
2026-03-14 09:56:26.673303 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.673307 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.673312 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.673316 | controller |       available"'
2026-03-14 09:56:26.673320 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.673324 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.673329 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.673333 | controller |     - '          exit 1'
2026-03-14 09:56:26.673337 | controller |     - '        fi'
2026-03-14 09:56:26.673341 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.673346 | controller |     - '        '
2026-03-14 09:56:26.673350 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.673354 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.673358 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.673363 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.673367 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.673371 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.673375 | controller |       -- \'
2026-03-14 09:56:26.673380 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.673384 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.673388 | controller |     - '    logger.go:42: 09:39:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.673392 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.673397 | controller |     - '    logger.go:42: 09:39:53 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.673401 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.673405 | controller |       found'
2026-03-14 09:56:26.673409 | controller |     - '    logger.go:42: 09:39:54 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.673414 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.673418 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.673422 | controller |     - '        '
2026-03-14 09:56:26.673426 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.673431 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.673435 | controller |     - '        '
2026-03-14 09:56:26.673439 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.673443 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.673449 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.673454 | controller |     - '          exit 1'
2026-03-14 09:56:26.673458 | controller |     - '        fi'
2026-03-14 09:56:26.673462 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.673466 | controller |     - '        '
2026-03-14 09:56:26.673471 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.673482 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.673487 | controller |       | base64 -d)'
2026-03-14 09:56:26.673491 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.673495 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.673499 | controller |     - '          exit 1'
2026-03-14 09:56:26.673504 | controller |     - '        fi'
2026-03-14 09:56:26.673508 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.673512 | controller |     - '        '
2026-03-14 09:56:26.673519 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.673523 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.673527 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.673531 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.673536 | controller |       --timeout=300s'
2026-03-14 09:56:26.673540 | controller |     - '        '
2026-03-14 09:56:26.673544 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.673548 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.673553 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.673557 | controller |     - '        '
2026-03-14 09:56:26.673561 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.673566 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.673570 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.673574 | controller |     - '        '
2026-03-14 09:56:26.673579 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.673583 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.673587 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.673592 | controller |       available"'
2026-03-14 09:56:26.673608 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.673613 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.673617 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.673621 | controller |     - '          exit 1'
2026-03-14 09:56:26.673626 | controller |     - '        fi'
2026-03-14 09:56:26.673630 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.673634 | controller |     - '        '
2026-03-14 09:56:26.673638 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.673643 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.673647 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.673651 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.673655 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.673662 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.673666 | controller |       -- \'
2026-03-14 09:56:26.673670 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.673675 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.673679 | controller |     - '    logger.go:42: 09:39:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.673683 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.673688 | controller |     - '    logger.go:42: 09:39:54 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.673692 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.673696 | controller |       found'
2026-03-14 09:56:26.673700 | controller |     - '    logger.go:42: 09:39:55 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.673704 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.673709 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.673713 | controller |     - '        '
2026-03-14 09:56:26.673717 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.673721 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.673726 | controller |     - '        '
2026-03-14 09:56:26.673730 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.673744 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.673748 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.673752 | controller |     - '          exit 1'
2026-03-14 09:56:26.673757 | controller |     - '        fi'
2026-03-14 09:56:26.673761 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.673765 | controller |     - '        '
2026-03-14 09:56:26.673770 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.673774 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.673785 | controller |       | base64 -d)'
2026-03-14 09:56:26.673790 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.673794 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.673798 | controller |     - '          exit 1'
2026-03-14 09:56:26.673803 | controller |     - '        fi'
2026-03-14 09:56:26.673807 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.673811 | controller |     - '        '
2026-03-14 09:56:26.673815 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.673820 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.673824 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.673828 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.673832 | controller |       --timeout=300s'
2026-03-14 09:56:26.673837 | controller |     - '        '
2026-03-14 09:56:26.673841 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.673845 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.673850 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.673854 | controller |     - '        '
2026-03-14 09:56:26.673858 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.673862 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.673868 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.673873 | controller |     - '        '
2026-03-14 09:56:26.673877 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.673881 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.673886 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.673890 | controller |       available"'
2026-03-14 09:56:26.673894 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.673898 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.673903 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.673907 | controller |     - '          exit 1'
2026-03-14 09:56:26.673911 | controller |     - '        fi'
2026-03-14 09:56:26.673915 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.673920 | controller |     - '        '
2026-03-14 09:56:26.673924 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.673928 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.673932 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.673937 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.673941 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.673945 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.673949 | controller |       -- \'
2026-03-14 09:56:26.673954 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.673964 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.673970 | controller |     - '    logger.go:42: 09:39:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.673982 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.673988 | controller |     - '    logger.go:42: 09:39:55 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.673994 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.674000 | controller |       found'
2026-03-14 09:56:26.674006 | controller |     - '    logger.go:42: 09:39:56 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.674010 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.674015 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.674019 | controller |     - '        '
2026-03-14 09:56:26.674023 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.674027 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.674031 | controller |     - '        '
2026-03-14 09:56:26.674036 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.674040 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.674044 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.674048 | controller |     - '          exit 1'
2026-03-14 09:56:26.674053 | controller |     - '        fi'
2026-03-14 09:56:26.674057 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.674061 | controller |     - '        '
2026-03-14 09:56:26.674066 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.674070 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.674074 | controller |       | base64 -d)'
2026-03-14 09:56:26.674091 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.674096 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.674100 | controller |     - '          exit 1'
2026-03-14 09:56:26.674105 | controller |     - '        fi'
2026-03-14 09:56:26.674109 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.674113 | controller |     - '        '
2026-03-14 09:56:26.674118 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.674122 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.674126 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.674130 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.674134 | controller |       --timeout=300s'
2026-03-14 09:56:26.674139 | controller |     - '        '
2026-03-14 09:56:26.674143 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.674147 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.674152 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.674156 | controller |     - '        '
2026-03-14 09:56:26.674160 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.674164 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.674169 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.674173 | controller |     - '        '
2026-03-14 09:56:26.674177 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.674181 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.674186 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.674190 | controller |       available"'
2026-03-14 09:56:26.674194 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.674198 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.674205 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.674210 | controller |     - '          exit 1'
2026-03-14 09:56:26.674214 | controller |     - '        fi'
2026-03-14 09:56:26.674218 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.674223 | controller |     - '        '
2026-03-14 09:56:26.674227 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.674231 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.674235 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.674240 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.674244 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.674248 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.674254 | controller |       -- \'
2026-03-14 09:56:26.674259 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.674264 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.674269 | controller |     - '    logger.go:42: 09:39:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.674275 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.674279 | controller |     - '    logger.go:42: 09:39:57 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.674283 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.674288 | controller |       found'
2026-03-14 09:56:26.674292 | controller |     - '    logger.go:42: 09:39:58 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.674296 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.674300 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.674305 | controller |     - '        '
2026-03-14 09:56:26.674309 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.674313 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.674317 | controller |     - '        '
2026-03-14 09:56:26.674321 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.674326 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.674330 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.674334 | controller |     - '          exit 1'
2026-03-14 09:56:26.674339 | controller |     - '        fi'
2026-03-14 09:56:26.674343 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.674347 | controller |     - '        '
2026-03-14 09:56:26.674351 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.674355 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.674360 | controller |       | base64 -d)'
2026-03-14 09:56:26.674364 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.674376 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.674381 | controller |     - '          exit 1'
2026-03-14 09:56:26.674385 | controller |     - '        fi'
2026-03-14 09:56:26.674389 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.674394 | controller |     - '        '
2026-03-14 09:56:26.674398 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.674402 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.674406 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.674411 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.674415 | controller |       --timeout=300s'
2026-03-14 09:56:26.674419 | controller |     - '        '
2026-03-14 09:56:26.674424 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.674428 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.674432 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.674436 | controller |     - '        '
2026-03-14 09:56:26.674440 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.674445 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.674449 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.674453 | controller |     - '        '
2026-03-14 09:56:26.674457 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.674462 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.674466 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.674470 | controller |       available"'
2026-03-14 09:56:26.674476 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.674480 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.674484 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.674489 | controller |     - '          exit 1'
2026-03-14 09:56:26.674493 | controller |     - '        fi'
2026-03-14 09:56:26.674497 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.674501 | controller |     - '        '
2026-03-14 09:56:26.674506 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.674510 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.674514 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.674518 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.674523 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.674527 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.674531 | controller |       -- \'
2026-03-14 09:56:26.674535 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.674539 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.674544 | controller |     - '    logger.go:42: 09:39:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.674548 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.674552 | controller |     - '    logger.go:42: 09:39:58 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.674556 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.674561 | controller |       found'
2026-03-14 09:56:26.674565 | controller |     - '    logger.go:42: 09:39:59 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.674569 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.674575 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.674580 | controller |     - '        '
2026-03-14 09:56:26.674584 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.674608 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.674616 | controller |     - '        '
2026-03-14 09:56:26.674688 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.674694 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.674698 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.674703 | controller |     - '          exit 1'
2026-03-14 09:56:26.674707 | controller |     - '        fi'
2026-03-14 09:56:26.674716 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.674720 | controller |     - '        '
2026-03-14 09:56:26.674725 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.674730 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.674735 | controller |       | base64 -d)'
2026-03-14 09:56:26.674740 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.674751 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.674756 | controller |     - '          exit 1'
2026-03-14 09:56:26.674760 | controller |     - '        fi'
2026-03-14 09:56:26.674764 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.674770 | controller |     - '        '
2026-03-14 09:56:26.674774 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.674778 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.674785 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.674789 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.674794 | controller |       --timeout=300s'
2026-03-14 09:56:26.674802 | controller |     - '        '
2026-03-14 09:56:26.674815 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.674820 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.674824 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.674829 | controller |     - '        '
2026-03-14 09:56:26.674834 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.674838 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.674843 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.674850 | controller |     - '        '
2026-03-14 09:56:26.674857 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.674861 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.674866 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.674870 | controller |       available"'
2026-03-14 09:56:26.674874 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.674879 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.674883 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.674887 | controller |     - '          exit 1'
2026-03-14 09:56:26.674891 | controller |     - '        fi'
2026-03-14 09:56:26.674896 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.674900 | controller |     - '        '
2026-03-14 09:56:26.674904 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.674908 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.674912 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.674917 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.674921 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.674925 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.674930 | controller |       -- \'
2026-03-14 09:56:26.674934 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.674938 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.674942 | controller |     - '    logger.go:42: 09:39:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.674947 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.674951 | controller |     - '    logger.go:42: 09:39:59 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.674955 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.674959 | controller |       found'
2026-03-14 09:56:26.674964 | controller |     - '    logger.go:42: 09:40:00 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.674979 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.674985 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.674992 | controller |     - '        '
2026-03-14 09:56:26.674996 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.675000 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.675005 | controller |     - '        '
2026-03-14 09:56:26.675009 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.675013 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.675017 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.675022 | controller |     - '          exit 1'
2026-03-14 09:56:26.675026 | controller |     - '        fi'
2026-03-14 09:56:26.675030 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.675034 | controller |     - '        '
2026-03-14 09:56:26.675038 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.675043 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.675047 | controller |       | base64 -d)'
2026-03-14 09:56:26.675052 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.675058 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.675064 | controller |     - '          exit 1'
2026-03-14 09:56:26.675083 | controller |     - '        fi'
2026-03-14 09:56:26.675089 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.675095 | controller |     - '        '
2026-03-14 09:56:26.675101 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.675106 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.675112 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.675120 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.675125 | controller |       --timeout=300s'
2026-03-14 09:56:26.675130 | controller |     - '        '
2026-03-14 09:56:26.675134 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.675138 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.675143 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.675147 | controller |     - '        '
2026-03-14 09:56:26.675151 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.675155 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.675160 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.675164 | controller |     - '        '
2026-03-14 09:56:26.675168 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.675173 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.675177 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.675181 | controller |       available"'
2026-03-14 09:56:26.675185 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.675190 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.675194 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.675198 | controller |     - '          exit 1'
2026-03-14 09:56:26.675202 | controller |     - '        fi'
2026-03-14 09:56:26.675206 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.675211 | controller |     - '        '
2026-03-14 09:56:26.675215 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.675221 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.675226 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.675230 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.675234 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.675238 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.675243 | controller |       -- \'
2026-03-14 09:56:26.675247 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.675251 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.675255 | controller |     - '    logger.go:42: 09:40:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.675260 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.675264 | controller |     - '    logger.go:42: 09:40:00 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.675268 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.675272 | controller |       found'
2026-03-14 09:56:26.675277 | controller |     - '    logger.go:42: 09:40:01 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.675281 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.675285 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.675289 | controller |     - '        '
2026-03-14 09:56:26.675294 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.675298 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.675302 | controller |     - '        '
2026-03-14 09:56:26.675306 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.675311 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.675315 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.675319 | controller |     - '          exit 1'
2026-03-14 09:56:26.675323 | controller |     - '        fi'
2026-03-14 09:56:26.675328 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.675332 | controller |     - '        '
2026-03-14 09:56:26.675343 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.675348 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.675352 | controller |       | base64 -d)'
2026-03-14 09:56:26.675356 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.675360 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.675365 | controller |     - '          exit 1'
2026-03-14 09:56:26.675369 | controller |     - '        fi'
2026-03-14 09:56:26.675381 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.675386 | controller |     - '        '
2026-03-14 09:56:26.675390 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.675395 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.675401 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.675405 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.675410 | controller |       --timeout=300s'
2026-03-14 09:56:26.675415 | controller |     - '        '
2026-03-14 09:56:26.675420 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.675425 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.675431 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.675435 | controller |     - '        '
2026-03-14 09:56:26.675440 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.675444 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.675448 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.675454 | controller |     - '        '
2026-03-14 09:56:26.675459 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.675463 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.675467 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.675471 | controller |       available"'
2026-03-14 09:56:26.675476 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.675480 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.675484 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.675488 | controller |     - '          exit 1'
2026-03-14 09:56:26.675492 | controller |     - '        fi'
2026-03-14 09:56:26.675498 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.675503 | controller |     - '        '
2026-03-14 09:56:26.675507 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.675511 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.675515 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.675520 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.675524 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.675528 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.675532 | controller |       -- \'
2026-03-14 09:56:26.675537 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.675541 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.675545 | controller |     - '    logger.go:42: 09:40:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.675549 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.675554 | controller |     - '    logger.go:42: 09:40:01 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.675558 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.675562 | controller |       found'
2026-03-14 09:56:26.675566 | controller |     - '    logger.go:42: 09:40:02 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.675571 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.675575 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.675579 | controller |     - '        '
2026-03-14 09:56:26.675583 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.675588 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.675592 | controller |     - '        '
2026-03-14 09:56:26.675617 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.675622 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.675626 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.675632 | controller |     - '          exit 1'
2026-03-14 09:56:26.675636 | controller |     - '        fi'
2026-03-14 09:56:26.675641 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.675645 | controller |     - '        '
2026-03-14 09:56:26.675649 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.675655 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.675660 | controller |       | base64 -d)'
2026-03-14 09:56:26.675664 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.675668 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.675672 | controller |     - '          exit 1'
2026-03-14 09:56:26.675677 | controller |     - '        fi'
2026-03-14 09:56:26.675681 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.675685 | controller |     - '        '
2026-03-14 09:56:26.675696 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.675701 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.675705 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.675709 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.675716 | controller |       --timeout=300s'
2026-03-14 09:56:26.675720 | controller |     - '        '
2026-03-14 09:56:26.675724 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.675730 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.675734 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.675739 | controller |     - '        '
2026-03-14 09:56:26.675743 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.675747 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.675751 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.675756 | controller |     - '        '
2026-03-14 09:56:26.675760 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.675764 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.675768 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.675773 | controller |       available"'
2026-03-14 09:56:26.675777 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.675781 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.675785 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.675790 | controller |     - '          exit 1'
2026-03-14 09:56:26.675794 | controller |     - '        fi'
2026-03-14 09:56:26.675800 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.675805 | controller |     - '        '
2026-03-14 09:56:26.675809 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.675813 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.675818 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.675822 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.675826 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.675832 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.675837 | controller |       -- \'
2026-03-14 09:56:26.675841 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.675845 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.675849 | controller |     - '    logger.go:42: 09:40:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.675854 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.675858 | controller |     - '    logger.go:42: 09:40:03 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.675862 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.675866 | controller |       found'
2026-03-14 09:56:26.675871 | controller |     - '    logger.go:42: 09:40:04 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.675875 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.675879 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.675883 | controller |     - '        '
2026-03-14 09:56:26.675888 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.675892 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.675896 | controller |     - '        '
2026-03-14 09:56:26.675900 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.675905 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.675909 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.675913 | controller |     - '          exit 1'
2026-03-14 09:56:26.675917 | controller |     - '        fi'
2026-03-14 09:56:26.675922 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.675926 | controller |     - '        '
2026-03-14 09:56:26.675930 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.675934 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.675939 | controller |       | base64 -d)'
2026-03-14 09:56:26.675943 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.675947 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.675951 | controller |     - '          exit 1'
2026-03-14 09:56:26.675956 | controller |     - '        fi'
2026-03-14 09:56:26.675960 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.675964 | controller |     - '        '
2026-03-14 09:56:26.675969 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.675978 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.675983 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.675987 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.675991 | controller |       --timeout=300s'
2026-03-14 09:56:26.675996 | controller |     - '        '
2026-03-14 09:56:26.676000 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.676004 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.676009 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.676013 | controller |     - '        '
2026-03-14 09:56:26.676017 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.676021 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.676026 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.676032 | controller |     - '        '
2026-03-14 09:56:26.676036 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.676040 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.676045 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.676049 | controller |       available"'
2026-03-14 09:56:26.676053 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.676057 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.676062 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.676066 | controller |     - '          exit 1'
2026-03-14 09:56:26.676070 | controller |     - '        fi'
2026-03-14 09:56:26.676074 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.676079 | controller |     - '        '
2026-03-14 09:56:26.676083 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.676087 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.676091 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.676096 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.676100 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.676104 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.676108 | controller |       -- \'
2026-03-14 09:56:26.676113 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.676117 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.676123 | controller |     - '    logger.go:42: 09:40:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.676129 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.676134 | controller |     - '    logger.go:42: 09:40:04 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.676141 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.676146 | controller |       found'
2026-03-14 09:56:26.676152 | controller |     - '    logger.go:42: 09:40:05 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.676158 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.676164 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.676169 | controller |     - '        '
2026-03-14 09:56:26.676173 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.676178 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.676182 | controller |     - '        '
2026-03-14 09:56:26.676186 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.676190 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.676195 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.676199 | controller |     - '          exit 1'
2026-03-14 09:56:26.676203 | controller |     - '        fi'
2026-03-14 09:56:26.676208 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.676212 | controller |     - '        '
2026-03-14 09:56:26.676216 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.676220 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.676225 | controller |       | base64 -d)'
2026-03-14 09:56:26.676229 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.676237 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.676241 | controller |     - '          exit 1'
2026-03-14 09:56:26.676245 | controller |     - '        fi'
2026-03-14 09:56:26.676250 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.676254 | controller |     - '        '
2026-03-14 09:56:26.676258 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.676270 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.676275 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.676279 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.676283 | controller |       --timeout=300s'
2026-03-14 09:56:26.676288 | controller |     - '        '
2026-03-14 09:56:26.676292 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.676296 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.676300 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.676305 | controller |     - '        '
2026-03-14 09:56:26.676309 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.676313 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.676317 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.676322 | controller |     - '        '
2026-03-14 09:56:26.676326 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.676333 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.676337 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.676341 | controller |       available"'
2026-03-14 09:56:26.676346 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.676350 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.676354 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.676358 | controller |     - '          exit 1'
2026-03-14 09:56:26.676362 | controller |     - '        fi'
2026-03-14 09:56:26.676367 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.676371 | controller |     - '        '
2026-03-14 09:56:26.676375 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.676379 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.676384 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.676388 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.676392 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.676396 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.676401 | controller |       -- \'
2026-03-14 09:56:26.676405 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.676409 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.676414 | controller |     - '    logger.go:42: 09:40:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.676418 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.676424 | controller |     - '    logger.go:42: 09:40:05 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.676428 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.676432 | controller |       found'
2026-03-14 09:56:26.676437 | controller |     - '    logger.go:42: 09:40:06 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.676441 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.676445 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.676450 | controller |     - '        '
2026-03-14 09:56:26.676454 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.676458 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.676462 | controller |     - '        '
2026-03-14 09:56:26.676467 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.676471 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.676475 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.676480 | controller |     - '          exit 1'
2026-03-14 09:56:26.676484 | controller |     - '        fi'
2026-03-14 09:56:26.676488 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.676492 | controller |     - '        '
2026-03-14 09:56:26.676496 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.676501 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.676505 | controller |       | base64 -d)'
2026-03-14 09:56:26.676509 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.676514 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.676518 | controller |     - '          exit 1'
2026-03-14 09:56:26.676522 | controller |     - '        fi'
2026-03-14 09:56:26.676526 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.676531 | controller |     - '        '
2026-03-14 09:56:26.676535 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.676539 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.676550 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.676554 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.676559 | controller |       --timeout=300s'
2026-03-14 09:56:26.676563 | controller |     - '        '
2026-03-14 09:56:26.676567 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.676572 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.676578 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.676582 | controller |     - '        '
2026-03-14 09:56:26.676586 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.676591 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.676606 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.676614 | controller |     - '        '
2026-03-14 09:56:26.676618 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.676623 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.676627 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.676631 | controller |       available"'
2026-03-14 09:56:26.676635 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.676642 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.676646 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.676650 | controller |     - '          exit 1'
2026-03-14 09:56:26.676655 | controller |     - '        fi'
2026-03-14 09:56:26.676659 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.676663 | controller |     - '        '
2026-03-14 09:56:26.676667 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.676672 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.676676 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.676680 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.676684 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.676689 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.676693 | controller |       -- \'
2026-03-14 09:56:26.676697 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.676701 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.676706 | controller |     - '    logger.go:42: 09:40:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.676710 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.676714 | controller |     - '    logger.go:42: 09:40:06 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.676718 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.676723 | controller |       found'
2026-03-14 09:56:26.676727 | controller |     - '    logger.go:42: 09:40:07 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.676731 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.676735 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.676740 | controller |     - '        '
2026-03-14 09:56:26.676744 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.676748 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.676752 | controller |     - '        '
2026-03-14 09:56:26.676757 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.676761 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.676765 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.676769 | controller |     - '          exit 1'
2026-03-14 09:56:26.676774 | controller |     - '        fi'
2026-03-14 09:56:26.676778 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.676782 | controller |     - '        '
2026-03-14 09:56:26.676786 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.676791 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.676795 | controller |       | base64 -d)'
2026-03-14 09:56:26.676799 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.676803 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.676808 | controller |     - '          exit 1'
2026-03-14 09:56:26.676812 | controller |     - '        fi'
2026-03-14 09:56:26.676816 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.676820 | controller |     - '        '
2026-03-14 09:56:26.676825 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.676829 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.676842 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.676847 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.676851 | controller |       --timeout=300s'
2026-03-14 09:56:26.676858 | controller |     - '        '
2026-03-14 09:56:26.676862 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.676866 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.676871 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.676875 | controller |     - '        '
2026-03-14 09:56:26.676880 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.676884 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.676888 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.676892 | controller |     - '        '
2026-03-14 09:56:26.676896 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.676901 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.676905 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.676911 | controller |       available"'
2026-03-14 09:56:26.676915 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.676919 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.676925 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.676929 | controller |     - '          exit 1'
2026-03-14 09:56:26.676934 | controller |     - '        fi'
2026-03-14 09:56:26.676938 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.676942 | controller |     - '        '
2026-03-14 09:56:26.676946 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.676951 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.676955 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.676959 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.676963 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.676968 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.676972 | controller |       -- \'
2026-03-14 09:56:26.676976 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.676980 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.676985 | controller |     - '    logger.go:42: 09:40:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.676989 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.676993 | controller |     - '    logger.go:42: 09:40:07 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.676997 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.677002 | controller |       found'
2026-03-14 09:56:26.677006 | controller |     - '    logger.go:42: 09:40:08 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.677010 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.677014 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.677019 | controller |     - '        '
2026-03-14 09:56:26.677023 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.677029 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.677033 | controller |     - '        '
2026-03-14 09:56:26.677037 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.677042 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.677046 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.677050 | controller |     - '          exit 1'
2026-03-14 09:56:26.677054 | controller |     - '        fi'
2026-03-14 09:56:26.677059 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.677065 | controller |     - '        '
2026-03-14 09:56:26.677069 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.677073 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.677079 | controller |       | base64 -d)'
2026-03-14 09:56:26.677083 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.677087 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.677092 | controller |     - '          exit 1'
2026-03-14 09:56:26.677096 | controller |     - '        fi'
2026-03-14 09:56:26.677100 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.677104 | controller |     - '        '
2026-03-14 09:56:26.677109 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.677113 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.677117 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.677128 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.677133 | controller |       --timeout=300s'
2026-03-14 09:56:26.677137 | controller |     - '        '
2026-03-14 09:56:26.677141 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.677146 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.677150 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.677154 | controller |     - '        '
2026-03-14 09:56:26.677158 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.677163 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.677167 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.677171 | controller |     - '        '
2026-03-14 09:56:26.677175 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.677180 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.677184 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.677188 | controller |       available"'
2026-03-14 09:56:26.677192 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.677196 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.677201 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.677205 | controller |     - '          exit 1'
2026-03-14 09:56:26.677209 | controller |     - '        fi'
2026-03-14 09:56:26.677213 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.677218 | controller |     - '        '
2026-03-14 09:56:26.677222 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.677228 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.677232 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.677236 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.677240 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.677245 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.677249 | controller |       -- \'
2026-03-14 09:56:26.677253 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.677257 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.677262 | controller |     - '    logger.go:42: 09:40:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.677266 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.677270 | controller |     - '    logger.go:42: 09:40:08 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.677274 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.677279 | controller |       found'
2026-03-14 09:56:26.677283 | controller |     - '    logger.go:42: 09:40:09 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.677287 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.677291 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.677296 | controller |     - '        '
2026-03-14 09:56:26.677300 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.677304 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.677308 | controller |     - '        '
2026-03-14 09:56:26.677313 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.677317 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.677321 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.677325 | controller |     - '          exit 1'
2026-03-14 09:56:26.677330 | controller |     - '        fi'
2026-03-14 09:56:26.677334 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.677338 | controller |     - '        '
2026-03-14 09:56:26.677342 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.677347 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.677351 | controller |       | base64 -d)'
2026-03-14 09:56:26.677355 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.677359 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.677363 | controller |     - '          exit 1'
2026-03-14 09:56:26.677368 | controller |     - '        fi'
2026-03-14 09:56:26.677372 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.677376 | controller |     - '        '
2026-03-14 09:56:26.677380 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.677385 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.677389 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.677393 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.677403 | controller |       --timeout=300s'
2026-03-14 09:56:26.677408 | controller |     - '        '
2026-03-14 09:56:26.677412 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.677416 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.677422 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.677427 | controller |     - '        '
2026-03-14 09:56:26.677431 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.677435 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.677440 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.677444 | controller |     - '        '
2026-03-14 09:56:26.677448 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.677452 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.677457 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.677461 | controller |       available"'
2026-03-14 09:56:26.677465 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.677470 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.677474 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.677478 | controller |     - '          exit 1'
2026-03-14 09:56:26.677482 | controller |     - '        fi'
2026-03-14 09:56:26.677487 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.677493 | controller |     - '        '
2026-03-14 09:56:26.677497 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.677501 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.677506 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.677513 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.677517 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.677522 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.677526 | controller |       -- \'
2026-03-14 09:56:26.677530 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.677534 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.677539 | controller |     - '    logger.go:42: 09:40:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.677543 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.677547 | controller |     - '    logger.go:42: 09:40:10 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.677551 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.677556 | controller |       found'
2026-03-14 09:56:26.677560 | controller |     - '    logger.go:42: 09:40:11 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.677564 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.677568 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.677573 | controller |     - '        '
2026-03-14 09:56:26.677577 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.677581 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.677585 | controller |     - '        '
2026-03-14 09:56:26.677589 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.677604 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.677630 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.677635 | controller |     - '          exit 1'
2026-03-14 09:56:26.677641 | controller |     - '        fi'
2026-03-14 09:56:26.677646 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.677650 | controller |     - '        '
2026-03-14 09:56:26.677654 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.677658 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.677663 | controller |       | base64 -d)'
2026-03-14 09:56:26.677668 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.677673 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.677679 | controller |     - '          exit 1'
2026-03-14 09:56:26.677683 | controller |     - '        fi'
2026-03-14 09:56:26.677687 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.677692 | controller |     - '        '
2026-03-14 09:56:26.677696 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.677700 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.677704 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.677709 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.677713 | controller |       --timeout=300s'
2026-03-14 09:56:26.677717 | controller |     - '        '
2026-03-14 09:56:26.677729 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.677734 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.677738 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.677742 | controller |     - '        '
2026-03-14 09:56:26.677747 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.677751 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.677755 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.677759 | controller |     - '        '
2026-03-14 09:56:26.677764 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.677768 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.677772 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.677777 | controller |       available"'
2026-03-14 09:56:26.677781 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.677785 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.677790 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.677794 | controller |     - '          exit 1'
2026-03-14 09:56:26.677798 | controller |     - '        fi'
2026-03-14 09:56:26.677802 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.677807 | controller |     - '        '
2026-03-14 09:56:26.677811 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.677815 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.677819 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.677824 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.677828 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.677832 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.677839 | controller |       -- \'
2026-03-14 09:56:26.677843 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.677847 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.677852 | controller |     - '    logger.go:42: 09:40:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.677856 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.677860 | controller |     - '    logger.go:42: 09:40:11 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.677864 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.677869 | controller |       found'
2026-03-14 09:56:26.677875 | controller |     - '    logger.go:42: 09:40:12 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.677879 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.677884 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.677888 | controller |     - '        '
2026-03-14 09:56:26.677892 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.677896 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.677901 | controller |     - '        '
2026-03-14 09:56:26.677905 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.677909 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.677913 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.677918 | controller |     - '          exit 1'
2026-03-14 09:56:26.677922 | controller |     - '        fi'
2026-03-14 09:56:26.677926 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.677930 | controller |     - '        '
2026-03-14 09:56:26.677935 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.677939 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.677943 | controller |       | base64 -d)'
2026-03-14 09:56:26.677949 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.677954 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.677958 | controller |     - '          exit 1'
2026-03-14 09:56:26.677962 | controller |     - '        fi'
2026-03-14 09:56:26.677966 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.677971 | controller |     - '        '
2026-03-14 09:56:26.677975 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.677979 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.677983 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.677988 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.677992 | controller |       --timeout=300s'
2026-03-14 09:56:26.677996 | controller |     - '        '
2026-03-14 09:56:26.678007 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.678011 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.678016 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.678020 | controller |     - '        '
2026-03-14 09:56:26.678024 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.678028 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.678033 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.678039 | controller |     - '        '
2026-03-14 09:56:26.678043 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.678048 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.678052 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.678056 | controller |       available"'
2026-03-14 09:56:26.678060 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.678065 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.678069 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.678073 | controller |     - '          exit 1'
2026-03-14 09:56:26.678077 | controller |     - '        fi'
2026-03-14 09:56:26.678082 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.678086 | controller |     - '        '
2026-03-14 09:56:26.678090 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.678096 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.678100 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.678105 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.678109 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.678113 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.678118 | controller |       -- \'
2026-03-14 09:56:26.678122 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.678126 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.678130 | controller |     - '    logger.go:42: 09:40:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.678135 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.678139 | controller |     - '    logger.go:42: 09:40:12 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.678143 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.678147 | controller |       found'
2026-03-14 09:56:26.678152 | controller |     - '    logger.go:42: 09:40:13 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.678156 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.678160 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.678164 | controller |     - '        '
2026-03-14 09:56:26.678169 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.678173 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.678177 | controller |     - '        '
2026-03-14 09:56:26.678181 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.678186 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.678192 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.678196 | controller |     - '          exit 1'
2026-03-14 09:56:26.678200 | controller |     - '        fi'
2026-03-14 09:56:26.678205 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.678209 | controller |     - '        '
2026-03-14 09:56:26.678213 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.678228 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.678232 | controller |       | base64 -d)'
2026-03-14 09:56:26.678236 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.678241 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.678246 | controller |     - '          exit 1'
2026-03-14 09:56:26.678251 | controller |     - '        fi'
2026-03-14 09:56:26.678255 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.678259 | controller |     - '        '
2026-03-14 09:56:26.678264 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.678268 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.678274 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.678279 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.678284 | controller |       --timeout=300s'
2026-03-14 09:56:26.678288 | controller |     - '        '
2026-03-14 09:56:26.678292 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.678304 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.678311 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.678317 | controller |     - '        '
2026-03-14 09:56:26.678322 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.678327 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.678331 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.678335 | controller |     - '        '
2026-03-14 09:56:26.678340 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.678344 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.678348 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.678352 | controller |       available"'
2026-03-14 09:56:26.678357 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.678361 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.678365 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.678370 | controller |     - '          exit 1'
2026-03-14 09:56:26.678374 | controller |     - '        fi'
2026-03-14 09:56:26.678378 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.678383 | controller |     - '        '
2026-03-14 09:56:26.678387 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.678391 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.678396 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.678400 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.678405 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.678411 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.678418 | controller |       -- \'
2026-03-14 09:56:26.678423 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.678428 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.678433 | controller |     - '    logger.go:42: 09:40:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.678438 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.678442 | controller |     - '    logger.go:42: 09:40:13 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.678449 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.678454 | controller |       found'
2026-03-14 09:56:26.678458 | controller |     - '    logger.go:42: 09:40:14 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.678462 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.678467 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.678471 | controller |     - '        '
2026-03-14 09:56:26.678475 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.678480 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.678484 | controller |     - '        '
2026-03-14 09:56:26.678488 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.678492 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.678497 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.678501 | controller |     - '          exit 1'
2026-03-14 09:56:26.678505 | controller |     - '        fi'
2026-03-14 09:56:26.678509 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.678514 | controller |     - '        '
2026-03-14 09:56:26.678518 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.678522 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.678526 | controller |       | base64 -d)'
2026-03-14 09:56:26.678531 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.678535 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.678539 | controller |     - '          exit 1'
2026-03-14 09:56:26.678543 | controller |     - '        fi'
2026-03-14 09:56:26.678548 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.678552 | controller |     - '        '
2026-03-14 09:56:26.678556 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.678560 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.678565 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.678569 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.678573 | controller |       --timeout=300s'
2026-03-14 09:56:26.678577 | controller |     - '        '
2026-03-14 09:56:26.678581 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.678592 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.678611 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.678617 | controller |     - '        '
2026-03-14 09:56:26.678622 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.678626 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.678630 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.678635 | controller |     - '        '
2026-03-14 09:56:26.678639 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.678643 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.678647 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.678652 | controller |       available"'
2026-03-14 09:56:26.678656 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.678660 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.678666 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.678671 | controller |     - '          exit 1'
2026-03-14 09:56:26.678675 | controller |     - '        fi'
2026-03-14 09:56:26.678679 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.678684 | controller |     - '        '
2026-03-14 09:56:26.678688 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.678692 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.678696 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.678701 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.678707 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.678711 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.678715 | controller |       -- \'
2026-03-14 09:56:26.678719 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.678724 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.678728 | controller |     - '    logger.go:42: 09:40:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.678732 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.678736 | controller |     - '    logger.go:42: 09:40:14 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.678741 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.678745 | controller |       found'
2026-03-14 09:56:26.678749 | controller |     - '    logger.go:42: 09:40:15 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.678753 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.678758 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.678762 | controller |     - '        '
2026-03-14 09:56:26.678766 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.678770 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.678774 | controller |     - '        '
2026-03-14 09:56:26.678779 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.678783 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.678787 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.678791 | controller |     - '          exit 1'
2026-03-14 09:56:26.678796 | controller |     - '        fi'
2026-03-14 09:56:26.678800 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.678804 | controller |     - '        '
2026-03-14 09:56:26.678808 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.678812 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.678817 | controller |       | base64 -d)'
2026-03-14 09:56:26.678821 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.678825 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.678829 | controller |     - '          exit 1'
2026-03-14 09:56:26.678834 | controller |     - '        fi'
2026-03-14 09:56:26.678838 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.678842 | controller |     - '        '
2026-03-14 09:56:26.678846 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.678851 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.678855 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.678862 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.678867 | controller |       --timeout=300s'
2026-03-14 09:56:26.678871 | controller |     - '        '
2026-03-14 09:56:26.678875 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.678879 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.678890 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.678895 | controller |     - '        '
2026-03-14 09:56:26.678899 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.678904 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.678908 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.678912 | controller |     - '        '
2026-03-14 09:56:26.678917 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.678921 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.678925 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.678929 | controller |       available"'
2026-03-14 09:56:26.678934 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.678938 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.678942 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.678946 | controller |     - '          exit 1'
2026-03-14 09:56:26.678951 | controller |     - '        fi'
2026-03-14 09:56:26.678955 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.678959 | controller |     - '        '
2026-03-14 09:56:26.678963 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.678968 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.678984 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.678991 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.678995 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.679000 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.679004 | controller |       -- \'
2026-03-14 09:56:26.679008 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.679012 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.679017 | controller |     - '    logger.go:42: 09:40:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.679021 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.679025 | controller |     - '    logger.go:42: 09:40:16 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.679029 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.679034 | controller |       found'
2026-03-14 09:56:26.679038 | controller |     - '    logger.go:42: 09:40:17 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.679042 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.679046 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.679051 | controller |     - '        '
2026-03-14 09:56:26.679055 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.679061 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.679066 | controller |     - '        '
2026-03-14 09:56:26.679070 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.679074 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.679078 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.679083 | controller |     - '          exit 1'
2026-03-14 09:56:26.679087 | controller |     - '        fi'
2026-03-14 09:56:26.679091 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.679095 | controller |     - '        '
2026-03-14 09:56:26.679100 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.679104 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.679108 | controller |       | base64 -d)'
2026-03-14 09:56:26.679113 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.679117 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.679121 | controller |     - '          exit 1'
2026-03-14 09:56:26.679125 | controller |     - '        fi'
2026-03-14 09:56:26.679129 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.679134 | controller |     - '        '
2026-03-14 09:56:26.679138 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.679142 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.679146 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.679152 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.679158 | controller |       --timeout=300s'
2026-03-14 09:56:26.679163 | controller |     - '        '
2026-03-14 09:56:26.679169 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.679174 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.679188 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.679195 | controller |     - '        '
2026-03-14 09:56:26.679201 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.679207 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.679213 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.679219 | controller |     - '        '
2026-03-14 09:56:26.679223 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.679228 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.679232 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.679236 | controller |       available"'
2026-03-14 09:56:26.679240 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.679245 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.679249 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.679253 | controller |     - '          exit 1'
2026-03-14 09:56:26.679257 | controller |     - '        fi'
2026-03-14 09:56:26.679262 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.679266 | controller |     - '        '
2026-03-14 09:56:26.679270 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.679274 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.679281 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.679285 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.679289 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.679293 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.679298 | controller |       -- \'
2026-03-14 09:56:26.679302 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.679306 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.679313 | controller |     - '    logger.go:42: 09:40:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.679317 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.679321 | controller |     - '    logger.go:42: 09:40:17 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.679326 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.679330 | controller |       found'
2026-03-14 09:56:26.679334 | controller |     - '    logger.go:42: 09:40:18 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.679339 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.679343 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.679347 | controller |     - '        '
2026-03-14 09:56:26.679351 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.679355 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.679360 | controller |     - '        '
2026-03-14 09:56:26.679364 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.679368 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.679372 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.679377 | controller |     - '          exit 1'
2026-03-14 09:56:26.679381 | controller |     - '        fi'
2026-03-14 09:56:26.679387 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.679393 | controller |     - '        '
2026-03-14 09:56:26.679399 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.679405 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.679409 | controller |       | base64 -d)'
2026-03-14 09:56:26.679413 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.679418 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.679422 | controller |     - '          exit 1'
2026-03-14 09:56:26.679426 | controller |     - '        fi'
2026-03-14 09:56:26.679430 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.679435 | controller |     - '        '
2026-03-14 09:56:26.679439 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.679443 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.679447 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.679452 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.679456 | controller |       --timeout=300s'
2026-03-14 09:56:26.679460 | controller |     - '        '
2026-03-14 09:56:26.679464 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.679469 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.679481 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.679488 | controller |     - '        '
2026-03-14 09:56:26.679492 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.679496 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.679501 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.679505 | controller |     - '        '
2026-03-14 09:56:26.679509 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.679513 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.679518 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.679522 | controller |       available"'
2026-03-14 09:56:26.679526 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.679531 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.679537 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.679541 | controller |     - '          exit 1'
2026-03-14 09:56:26.679545 | controller |     - '        fi'
2026-03-14 09:56:26.679550 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.679554 | controller |     - '        '
2026-03-14 09:56:26.679558 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.679562 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.679567 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.679571 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.679575 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.679581 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.679586 | controller |       -- \'
2026-03-14 09:56:26.679590 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.679605 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.679613 | controller |     - '    logger.go:42: 09:40:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.679617 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.679622 | controller |     - '    logger.go:42: 09:40:18 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.679626 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.679630 | controller |       found'
2026-03-14 09:56:26.679635 | controller |     - '    logger.go:42: 09:40:19 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.679639 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.679643 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.679647 | controller |     - '        '
2026-03-14 09:56:26.679652 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.679656 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.679660 | controller |     - '        '
2026-03-14 09:56:26.679664 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.679669 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.679673 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.679677 | controller |     - '          exit 1'
2026-03-14 09:56:26.679681 | controller |     - '        fi'
2026-03-14 09:56:26.679688 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.679692 | controller |     - '        '
2026-03-14 09:56:26.679697 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.679701 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.679705 | controller |       | base64 -d)'
2026-03-14 09:56:26.679709 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.679714 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.679718 | controller |     - '          exit 1'
2026-03-14 09:56:26.679722 | controller |     - '        fi'
2026-03-14 09:56:26.679726 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.679731 | controller |     - '        '
2026-03-14 09:56:26.679735 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.679739 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.679744 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.679748 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.679752 | controller |       --timeout=300s'
2026-03-14 09:56:26.679756 | controller |     - '        '
2026-03-14 09:56:26.679761 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.679767 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.679771 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.679776 | controller |     - '        '
2026-03-14 09:56:26.679787 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.679791 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.679796 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.679800 | controller |     - '        '
2026-03-14 09:56:26.679804 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.679809 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.679813 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.679817 | controller |       available"'
2026-03-14 09:56:26.679822 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.679826 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.679830 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.679834 | controller |     - '          exit 1'
2026-03-14 09:56:26.679839 | controller |     - '        fi'
2026-03-14 09:56:26.679843 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.679847 | controller |     - '        '
2026-03-14 09:56:26.679851 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.679855 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.679860 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.679864 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.679868 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.679872 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.679877 | controller |       -- \'
2026-03-14 09:56:26.679883 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.679887 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.679891 | controller |     - '    logger.go:42: 09:40:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.679896 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.679900 | controller |     - '    logger.go:42: 09:40:19 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.679906 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.679911 | controller |       found'
2026-03-14 09:56:26.679915 | controller |     - '    logger.go:42: 09:40:20 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.679919 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.679923 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.679928 | controller |     - '        '
2026-03-14 09:56:26.679932 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.679936 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.679940 | controller |     - '        '
2026-03-14 09:56:26.679945 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.679949 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.679953 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.679957 | controller |     - '          exit 1'
2026-03-14 09:56:26.679961 | controller |     - '        fi'
2026-03-14 09:56:26.679966 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.679970 | controller |     - '        '
2026-03-14 09:56:26.679974 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.679978 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.679982 | controller |       | base64 -d)'
2026-03-14 09:56:26.679987 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.679991 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.679995 | controller |     - '          exit 1'
2026-03-14 09:56:26.679999 | controller |     - '        fi'
2026-03-14 09:56:26.680004 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.680008 | controller |     - '        '
2026-03-14 09:56:26.680012 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.680016 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.680021 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.680025 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.680029 | controller |       --timeout=300s'
2026-03-14 09:56:26.680033 | controller |     - '        '
2026-03-14 09:56:26.680038 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.680042 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.680046 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.680050 | controller |     - '        '
2026-03-14 09:56:26.680061 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.680066 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.680070 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.680076 | controller |     - '        '
2026-03-14 09:56:26.680082 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.680086 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.680091 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.680095 | controller |       available"'
2026-03-14 09:56:26.680099 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.680103 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.680108 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.680112 | controller |     - '          exit 1'
2026-03-14 09:56:26.680116 | controller |     - '        fi'
2026-03-14 09:56:26.680120 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.680125 | controller |     - '        '
2026-03-14 09:56:26.680129 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.680133 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.680137 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.680142 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.680146 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.680150 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.680154 | controller |       -- \'
2026-03-14 09:56:26.680158 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.680163 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.680167 | controller |     - '    logger.go:42: 09:40:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.680171 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.680176 | controller |     - '    logger.go:42: 09:40:20 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.680180 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.680184 | controller |       found'
2026-03-14 09:56:26.680188 | controller |     - '    logger.go:42: 09:40:21 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.680192 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.680197 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.680201 | controller |     - '        '
2026-03-14 09:56:26.680205 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.680209 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.680214 | controller |     - '        '
2026-03-14 09:56:26.680218 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.680222 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.680226 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.680231 | controller |     - '          exit 1'
2026-03-14 09:56:26.680235 | controller |     - '        fi'
2026-03-14 09:56:26.680239 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.680243 | controller |     - '        '
2026-03-14 09:56:26.680248 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.680252 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.680257 | controller |       | base64 -d)'
2026-03-14 09:56:26.680262 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.680266 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.680273 | controller |     - '          exit 1'
2026-03-14 09:56:26.680278 | controller |     - '        fi'
2026-03-14 09:56:26.680282 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.680286 | controller |     - '        '
2026-03-14 09:56:26.680291 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.680295 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.680299 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.680303 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.680307 | controller |       --timeout=300s'
2026-03-14 09:56:26.680312 | controller |     - '        '
2026-03-14 09:56:26.680316 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.680320 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.680324 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.680329 | controller |     - '        '
2026-03-14 09:56:26.680333 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.680343 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.680348 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.680352 | controller |     - '        '
2026-03-14 09:56:26.680357 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.680361 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.680365 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.680369 | controller |       available"'
2026-03-14 09:56:26.680374 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.680378 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.680382 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.680386 | controller |     - '          exit 1'
2026-03-14 09:56:26.680391 | controller |     - '        fi'
2026-03-14 09:56:26.680395 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.680399 | controller |     - '        '
2026-03-14 09:56:26.680403 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.680407 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.680412 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.680416 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.680420 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.680424 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.680429 | controller |       -- \'
2026-03-14 09:56:26.680433 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.680437 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.680441 | controller |     - '    logger.go:42: 09:40:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.680445 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.680450 | controller |     - '    logger.go:42: 09:40:21 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.680456 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.680464 | controller |       found'
2026-03-14 09:56:26.680470 | controller |     - '    logger.go:42: 09:40:22 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.680476 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.680480 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.680485 | controller |     - '        '
2026-03-14 09:56:26.680489 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.680493 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.680497 | controller |     - '        '
2026-03-14 09:56:26.680501 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.680506 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.680510 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.680514 | controller |     - '          exit 1'
2026-03-14 09:56:26.680518 | controller |     - '        fi'
2026-03-14 09:56:26.680523 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.680527 | controller |     - '        '
2026-03-14 09:56:26.680531 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.680535 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.680540 | controller |       | base64 -d)'
2026-03-14 09:56:26.680544 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.680548 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.680552 | controller |     - '          exit 1'
2026-03-14 09:56:26.680557 | controller |     - '        fi'
2026-03-14 09:56:26.680561 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.680565 | controller |     - '        '
2026-03-14 09:56:26.680569 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.680573 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.680578 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.680582 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.680586 | controller |       --timeout=300s'
2026-03-14 09:56:26.680590 | controller |     - '        '
2026-03-14 09:56:26.680604 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.680611 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.680615 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.680619 | controller |     - '        '
2026-03-14 09:56:26.680624 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.680635 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.680642 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.680646 | controller |     - '        '
2026-03-14 09:56:26.680651 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.680655 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.680659 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.680663 | controller |       available"'
2026-03-14 09:56:26.680668 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.680672 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.680676 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.680683 | controller |     - '          exit 1'
2026-03-14 09:56:26.680687 | controller |     - '        fi'
2026-03-14 09:56:26.680691 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.680696 | controller |     - '        '
2026-03-14 09:56:26.680700 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.680704 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.680708 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.680712 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.680717 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.680721 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.680725 | controller |       -- \'
2026-03-14 09:56:26.680729 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.680734 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.680738 | controller |     - '    logger.go:42: 09:40:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.680742 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.680746 | controller |     - '    logger.go:42: 09:40:22 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.680750 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.680755 | controller |       found'
2026-03-14 09:56:26.680759 | controller |     - '    logger.go:42: 09:40:23 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.680763 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.680767 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.680772 | controller |     - '        '
2026-03-14 09:56:26.680776 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.680780 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.680784 | controller |     - '        '
2026-03-14 09:56:26.680788 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.680793 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.680797 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.680801 | controller |     - '          exit 1'
2026-03-14 09:56:26.680805 | controller |     - '        fi'
2026-03-14 09:56:26.680810 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.680814 | controller |     - '        '
2026-03-14 09:56:26.680818 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.680824 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.680829 | controller |       | base64 -d)'
2026-03-14 09:56:26.680833 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.680837 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.680841 | controller |     - '          exit 1'
2026-03-14 09:56:26.680846 | controller |     - '        fi'
2026-03-14 09:56:26.680850 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.680854 | controller |     - '        '
2026-03-14 09:56:26.680858 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.680863 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.680867 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.680871 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.680878 | controller |       --timeout=300s'
2026-03-14 09:56:26.680882 | controller |     - '        '
2026-03-14 09:56:26.680886 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.680891 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.680895 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.680899 | controller |     - '        '
2026-03-14 09:56:26.680904 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.680908 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.680919 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.680923 | controller |     - '        '
2026-03-14 09:56:26.680928 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.680932 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.680936 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.680940 | controller |       available"'
2026-03-14 09:56:26.680945 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.680949 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.680953 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.680958 | controller |     - '          exit 1'
2026-03-14 09:56:26.680962 | controller |     - '        fi'
2026-03-14 09:56:26.680966 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.680971 | controller |     - '        '
2026-03-14 09:56:26.680975 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.680979 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.680983 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.680988 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.680992 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.680996 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.681000 | controller |       -- \'
2026-03-14 09:56:26.681005 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.681009 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.681013 | controller |     - '    logger.go:42: 09:40:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.681017 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.681022 | controller |     - '    logger.go:42: 09:40:24 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.681026 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.681030 | controller |       found'
2026-03-14 09:56:26.681035 | controller |     - '    logger.go:42: 09:40:25 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.681041 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.681045 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.681049 | controller |     - '        '
2026-03-14 09:56:26.681054 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.681058 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.681064 | controller |     - '        '
2026-03-14 09:56:26.681068 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.681072 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.681077 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.681081 | controller |     - '          exit 1'
2026-03-14 09:56:26.681085 | controller |     - '        fi'
2026-03-14 09:56:26.681090 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.681095 | controller |     - '        '
2026-03-14 09:56:26.681100 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.681104 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.681110 | controller |       | base64 -d)'
2026-03-14 09:56:26.681114 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.681118 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.681123 | controller |     - '          exit 1'
2026-03-14 09:56:26.681127 | controller |     - '        fi'
2026-03-14 09:56:26.681133 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.681137 | controller |     - '        '
2026-03-14 09:56:26.681141 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.681146 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.681150 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.681154 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.681158 | controller |       --timeout=300s'
2026-03-14 09:56:26.681162 | controller |     - '        '
2026-03-14 09:56:26.681167 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.681171 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.681175 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.681179 | controller |     - '        '
2026-03-14 09:56:26.681184 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.681188 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.681199 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.681203 | controller |     - '        '
2026-03-14 09:56:26.681208 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.681212 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.681216 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.681220 | controller |       available"'
2026-03-14 09:56:26.681225 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.681229 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.681233 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.681237 | controller |     - '          exit 1'
2026-03-14 09:56:26.681242 | controller |     - '        fi'
2026-03-14 09:56:26.681246 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.681250 | controller |     - '        '
2026-03-14 09:56:26.681254 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.681258 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.681263 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.681269 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.681273 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.681277 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.681282 | controller |       -- \'
2026-03-14 09:56:26.681286 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.681290 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.681294 | controller |     - '    logger.go:42: 09:40:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.681299 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.681303 | controller |     - '    logger.go:42: 09:40:25 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.681307 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.681311 | controller |       found'
2026-03-14 09:56:26.681315 | controller |     - '    logger.go:42: 09:40:26 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.681320 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.681324 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.681328 | controller |     - '        '
2026-03-14 09:56:26.681333 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.681337 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.681341 | controller |     - '        '
2026-03-14 09:56:26.681345 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.681349 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.681354 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.681358 | controller |     - '          exit 1'
2026-03-14 09:56:26.681362 | controller |     - '        fi'
2026-03-14 09:56:26.681366 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.681371 | controller |     - '        '
2026-03-14 09:56:26.681375 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.681379 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.681383 | controller |       | base64 -d)'
2026-03-14 09:56:26.681387 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.681392 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.681396 | controller |     - '          exit 1'
2026-03-14 09:56:26.681400 | controller |     - '        fi'
2026-03-14 09:56:26.681404 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.681408 | controller |     - '        '
2026-03-14 09:56:26.681413 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.681417 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.681421 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.681425 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.681429 | controller |       --timeout=300s'
2026-03-14 09:56:26.681434 | controller |     - '        '
2026-03-14 09:56:26.681438 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.681442 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.681446 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.681452 | controller |     - '        '
2026-03-14 09:56:26.681456 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.681461 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.681465 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.681475 | controller |     - '        '
2026-03-14 09:56:26.681480 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.681484 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.681488 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.681492 | controller |       available"'
2026-03-14 09:56:26.681497 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.681501 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.681505 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.681509 | controller |     - '          exit 1'
2026-03-14 09:56:26.681513 | controller |     - '        fi'
2026-03-14 09:56:26.681518 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.681522 | controller |     - '        '
2026-03-14 09:56:26.681526 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.681530 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.681536 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.681542 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.681548 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.681553 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.681558 | controller |       -- \'
2026-03-14 09:56:26.681562 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.681569 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.681573 | controller |     - '    logger.go:42: 09:40:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.681577 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.681581 | controller |     - '    logger.go:42: 09:40:26 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.681586 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.681590 | controller |       found'
2026-03-14 09:56:26.681605 | controller |     - '    logger.go:42: 09:40:27 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.681611 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.681615 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.681619 | controller |     - '        '
2026-03-14 09:56:26.681624 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.681631 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.681636 | controller |     - '        '
2026-03-14 09:56:26.681640 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.681644 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.681648 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.681653 | controller |     - '          exit 1'
2026-03-14 09:56:26.681657 | controller |     - '        fi'
2026-03-14 09:56:26.681661 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.681668 | controller |     - '        '
2026-03-14 09:56:26.681672 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.681676 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.681680 | controller |       | base64 -d)'
2026-03-14 09:56:26.681685 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.681689 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.681693 | controller |     - '          exit 1'
2026-03-14 09:56:26.681697 | controller |     - '        fi'
2026-03-14 09:56:26.681702 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.681706 | controller |     - '        '
2026-03-14 09:56:26.681710 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.681714 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.681718 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.681723 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.681727 | controller |       --timeout=300s'
2026-03-14 09:56:26.681731 | controller |     - '        '
2026-03-14 09:56:26.681735 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.681740 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.681744 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.681748 | controller |     - '        '
2026-03-14 09:56:26.681752 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.681757 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.681761 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.681765 | controller |     - '        '
2026-03-14 09:56:26.681777 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.681781 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.681786 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.681790 | controller |       available"'
2026-03-14 09:56:26.681794 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.681799 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.681803 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.681807 | controller |     - '          exit 1'
2026-03-14 09:56:26.681811 | controller |     - '        fi'
2026-03-14 09:56:26.681816 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.681820 | controller |     - '        '
2026-03-14 09:56:26.681824 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.681828 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.681833 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.681837 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.681841 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.681845 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.681850 | controller |       -- \'
2026-03-14 09:56:26.681854 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.681860 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.681864 | controller |     - '    logger.go:42: 09:40:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.681869 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.681873 | controller |     - '    logger.go:42: 09:40:27 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.681877 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.681882 | controller |       found'
2026-03-14 09:56:26.681886 | controller |     - '    logger.go:42: 09:40:28 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.681890 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.681896 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.681901 | controller |     - '        '
2026-03-14 09:56:26.681905 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.681909 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.681914 | controller |     - '        '
2026-03-14 09:56:26.681918 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.681922 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.681926 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.681931 | controller |     - '          exit 1'
2026-03-14 09:56:26.681935 | controller |     - '        fi'
2026-03-14 09:56:26.681939 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.681944 | controller |     - '        '
2026-03-14 09:56:26.681948 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.681952 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.681956 | controller |       | base64 -d)'
2026-03-14 09:56:26.681960 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.681965 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.681969 | controller |     - '          exit 1'
2026-03-14 09:56:26.681973 | controller |     - '        fi'
2026-03-14 09:56:26.681977 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.681981 | controller |     - '        '
2026-03-14 09:56:26.681986 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.681990 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.681994 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.681998 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.682004 | controller |       --timeout=300s'
2026-03-14 09:56:26.682010 | controller |     - '        '
2026-03-14 09:56:26.682015 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.682021 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.682027 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.682033 | controller |     - '        '
2026-03-14 09:56:26.682038 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.682044 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.682049 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.682055 | controller |     - '        '
2026-03-14 09:56:26.682060 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.682077 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.682083 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.682089 | controller |       available"'
2026-03-14 09:56:26.682094 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.682099 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.682105 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.682109 | controller |     - '          exit 1'
2026-03-14 09:56:26.682114 | controller |     - '        fi'
2026-03-14 09:56:26.682118 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.682122 | controller |     - '        '
2026-03-14 09:56:26.682163 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.682168 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.682173 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.682177 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.682181 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.682185 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.682190 | controller |       -- \'
2026-03-14 09:56:26.682194 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.682198 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.682202 | controller |     - '    logger.go:42: 09:40:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.682207 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.682211 | controller |     - '    logger.go:42: 09:40:28 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.682215 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.682219 | controller |       found'
2026-03-14 09:56:26.682223 | controller |     - '    logger.go:42: 09:40:29 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.682228 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.682232 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.682236 | controller |     - '        '
2026-03-14 09:56:26.682240 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.682245 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.682251 | controller |     - '        '
2026-03-14 09:56:26.682256 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.682260 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.682264 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.682269 | controller |     - '          exit 1'
2026-03-14 09:56:26.682275 | controller |     - '        fi'
2026-03-14 09:56:26.682279 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.682283 | controller |     - '        '
2026-03-14 09:56:26.682287 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.682292 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.682296 | controller |       | base64 -d)'
2026-03-14 09:56:26.682300 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.682304 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.682308 | controller |     - '          exit 1'
2026-03-14 09:56:26.682315 | controller |     - '        fi'
2026-03-14 09:56:26.682319 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.682323 | controller |     - '        '
2026-03-14 09:56:26.682328 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.682332 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.682336 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.682340 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.682344 | controller |       --timeout=300s'
2026-03-14 09:56:26.682349 | controller |     - '        '
2026-03-14 09:56:26.682353 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.682357 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.682361 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.682365 | controller |     - '        '
2026-03-14 09:56:26.682370 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.682374 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.682378 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.682382 | controller |     - '        '
2026-03-14 09:56:26.682386 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.682399 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.682404 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.682408 | controller |       available"'
2026-03-14 09:56:26.682412 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.682417 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.682421 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.682425 | controller |     - '          exit 1'
2026-03-14 09:56:26.682429 | controller |     - '        fi'
2026-03-14 09:56:26.682434 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.682438 | controller |     - '        '
2026-03-14 09:56:26.682442 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.682446 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.682451 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.682455 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.682459 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.682463 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.682470 | controller |       -- \'
2026-03-14 09:56:26.682474 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.682478 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.682482 | controller |     - '    logger.go:42: 09:40:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.682487 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.682491 | controller |     - '    logger.go:42: 09:40:29 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.682495 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.682501 | controller |       found'
2026-03-14 09:56:26.682505 | controller |     - '    logger.go:42: 09:40:30 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.682510 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.682514 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.682518 | controller |     - '        '
2026-03-14 09:56:26.682522 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.682527 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.682531 | controller |     - '        '
2026-03-14 09:56:26.682535 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.682539 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.682544 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.682548 | controller |     - '          exit 1'
2026-03-14 09:56:26.682552 | controller |     - '        fi'
2026-03-14 09:56:26.682556 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.682560 | controller |     - '        '
2026-03-14 09:56:26.682565 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.682569 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.682573 | controller |       | base64 -d)'
2026-03-14 09:56:26.682577 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.682582 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.682586 | controller |     - '          exit 1'
2026-03-14 09:56:26.682590 | controller |     - '        fi'
2026-03-14 09:56:26.682605 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.682614 | controller |     - '        '
2026-03-14 09:56:26.682620 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.682626 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.682640 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.682644 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.682649 | controller |       --timeout=300s'
2026-03-14 09:56:26.682653 | controller |     - '        '
2026-03-14 09:56:26.682657 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.682661 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.682666 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.682672 | controller |     - '        '
2026-03-14 09:56:26.682676 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.682681 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.682685 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.682691 | controller |     - '        '
2026-03-14 09:56:26.682695 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.682699 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.682712 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.682717 | controller |       available"'
2026-03-14 09:56:26.682721 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.682726 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.682730 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.682736 | controller |     - '          exit 1'
2026-03-14 09:56:26.682741 | controller |     - '        fi'
2026-03-14 09:56:26.682745 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.682749 | controller |     - '        '
2026-03-14 09:56:26.682753 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.682758 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.682762 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.682766 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.682770 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.682774 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.682779 | controller |       -- \'
2026-03-14 09:56:26.682783 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.682787 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.682791 | controller |     - '    logger.go:42: 09:40:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.682796 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.682800 | controller |     - '    logger.go:42: 09:40:31 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.682804 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.682808 | controller |       found'
2026-03-14 09:56:26.682813 | controller |     - '    logger.go:42: 09:40:32 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.682817 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.682821 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.682825 | controller |     - '        '
2026-03-14 09:56:26.682830 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.682834 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.682838 | controller |     - '        '
2026-03-14 09:56:26.682842 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.682849 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.682853 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.682857 | controller |     - '          exit 1'
2026-03-14 09:56:26.682861 | controller |     - '        fi'
2026-03-14 09:56:26.682866 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.682870 | controller |     - '        '
2026-03-14 09:56:26.682874 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.682878 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.682882 | controller |       | base64 -d)'
2026-03-14 09:56:26.682887 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.682891 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.682895 | controller |     - '          exit 1'
2026-03-14 09:56:26.682899 | controller |     - '        fi'
2026-03-14 09:56:26.682904 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.682908 | controller |     - '        '
2026-03-14 09:56:26.682912 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.682916 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.682920 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.682925 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.682932 | controller |       --timeout=300s'
2026-03-14 09:56:26.682936 | controller |     - '        '
2026-03-14 09:56:26.682941 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.682945 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.682949 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.682954 | controller |     - '        '
2026-03-14 09:56:26.682958 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.682962 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.682966 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.682971 | controller |     - '        '
2026-03-14 09:56:26.682991 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.682995 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.683000 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.683010 | controller |       available"'
2026-03-14 09:56:26.683015 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.683019 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.683023 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.683027 | controller |     - '          exit 1'
2026-03-14 09:56:26.683032 | controller |     - '        fi'
2026-03-14 09:56:26.683036 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.683040 | controller |     - '        '
2026-03-14 09:56:26.683045 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.683049 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.683053 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.683057 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.683062 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.683066 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.683070 | controller |       -- \'
2026-03-14 09:56:26.683074 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.683079 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.683083 | controller |     - '    logger.go:42: 09:40:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.683087 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.683091 | controller |     - '    logger.go:42: 09:40:32 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.683095 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.683100 | controller |       found'
2026-03-14 09:56:26.683104 | controller |     - '    logger.go:42: 09:40:33 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.683108 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.683112 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.683117 | controller |     - '        '
2026-03-14 09:56:26.683121 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.683125 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.683131 | controller |     - '        '
2026-03-14 09:56:26.683136 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.683140 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.683144 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.683148 | controller |     - '          exit 1'
2026-03-14 09:56:26.683153 | controller |     - '        fi'
2026-03-14 09:56:26.683157 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.683161 | controller |     - '        '
2026-03-14 09:56:26.683165 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.683170 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.683174 | controller |       | base64 -d)'
2026-03-14 09:56:26.683178 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.683182 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.683186 | controller |     - '          exit 1'
2026-03-14 09:56:26.683191 | controller |     - '        fi'
2026-03-14 09:56:26.683195 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.683199 | controller |     - '        '
2026-03-14 09:56:26.683204 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.683208 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.683212 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.683216 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.683220 | controller |       --timeout=300s'
2026-03-14 09:56:26.683225 | controller |     - '        '
2026-03-14 09:56:26.683229 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.683233 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.683237 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.683242 | controller |     - '        '
2026-03-14 09:56:26.683246 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.683250 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.683254 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.683259 | controller |     - '        '
2026-03-14 09:56:26.683263 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.683267 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.683271 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.683276 | controller |       available"'
2026-03-14 09:56:26.683286 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.683291 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.683295 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.683299 | controller |     - '          exit 1'
2026-03-14 09:56:26.683307 | controller |     - '        fi'
2026-03-14 09:56:26.683311 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.683316 | controller |     - '        '
2026-03-14 09:56:26.683320 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.683324 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.683328 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.683334 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.683339 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.683343 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.683347 | controller |       -- \'
2026-03-14 09:56:26.683352 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.683356 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.683360 | controller |     - '    logger.go:42: 09:40:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.683364 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.683369 | controller |     - '    logger.go:42: 09:40:33 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.683373 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.683377 | controller |       found'
2026-03-14 09:56:26.683381 | controller |     - '    logger.go:42: 09:40:34 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.683385 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.683390 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.683394 | controller |     - '        '
2026-03-14 09:56:26.683398 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.683402 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.683407 | controller |     - '        '
2026-03-14 09:56:26.683411 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.683415 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.683419 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.683424 | controller |     - '          exit 1'
2026-03-14 09:56:26.683428 | controller |     - '        fi'
2026-03-14 09:56:26.683432 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.683436 | controller |     - '        '
2026-03-14 09:56:26.683441 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.683447 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.683451 | controller |       | base64 -d)'
2026-03-14 09:56:26.683455 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.683459 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.683464 | controller |     - '          exit 1'
2026-03-14 09:56:26.683468 | controller |     - '        fi'
2026-03-14 09:56:26.683472 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.683476 | controller |     - '        '
2026-03-14 09:56:26.683481 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.683485 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.683491 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.683496 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.683500 | controller |       --timeout=300s'
2026-03-14 09:56:26.683504 | controller |     - '        '
2026-03-14 09:56:26.683508 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.683512 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.683517 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.683521 | controller |     - '        '
2026-03-14 09:56:26.683527 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.683531 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.683535 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.683540 | controller |     - '        '
2026-03-14 09:56:26.683544 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.683548 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.683552 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.683557 | controller |       available"'
2026-03-14 09:56:26.683561 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.683572 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.683576 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.683581 | controller |     - '          exit 1'
2026-03-14 09:56:26.683585 | controller |     - '        fi'
2026-03-14 09:56:26.683589 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.683593 | controller |     - '        '
2026-03-14 09:56:26.683610 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.683615 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.683619 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.683623 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.683628 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.683632 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.683636 | controller |       -- \'
2026-03-14 09:56:26.683640 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.683645 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.683649 | controller |     - '    logger.go:42: 09:40:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.683653 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.683658 | controller |     - '    logger.go:42: 09:40:34 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.683662 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.683666 | controller |       found'
2026-03-14 09:56:26.683670 | controller |     - '    logger.go:42: 09:40:35 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.683675 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.683679 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.683683 | controller |     - '        '
2026-03-14 09:56:26.683687 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.683693 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.683699 | controller |     - '        '
2026-03-14 09:56:26.683705 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.683711 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.683715 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.683719 | controller |     - '          exit 1'
2026-03-14 09:56:26.683724 | controller |     - '        fi'
2026-03-14 09:56:26.683728 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.683732 | controller |     - '        '
2026-03-14 09:56:26.683736 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.683743 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.683747 | controller |       | base64 -d)'
2026-03-14 09:56:26.683752 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.683756 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.683760 | controller |     - '          exit 1'
2026-03-14 09:56:26.683764 | controller |     - '        fi'
2026-03-14 09:56:26.683768 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.683773 | controller |     - '        '
2026-03-14 09:56:26.683777 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.683781 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.683785 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.683790 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.683794 | controller |       --timeout=300s'
2026-03-14 09:56:26.683798 | controller |     - '        '
2026-03-14 09:56:26.683803 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.683809 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.683815 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.683819 | controller |     - '        '
2026-03-14 09:56:26.683823 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.683828 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.683832 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.683836 | controller |     - '        '
2026-03-14 09:56:26.683840 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.683844 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.683849 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.683853 | controller |       available"'
2026-03-14 09:56:26.683857 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.683869 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.683874 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.683878 | controller |     - '          exit 1'
2026-03-14 09:56:26.683882 | controller |     - '        fi'
2026-03-14 09:56:26.683886 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.683891 | controller |     - '        '
2026-03-14 09:56:26.683895 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.683899 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.683903 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.683907 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.683912 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.683916 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.683920 | controller |       -- \'
2026-03-14 09:56:26.683924 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.683930 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.683935 | controller |     - '    logger.go:42: 09:40:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.683939 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.683943 | controller |     - '    logger.go:42: 09:40:35 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.683948 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.683952 | controller |       found'
2026-03-14 09:56:26.683956 | controller |     - '    logger.go:42: 09:40:36 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.683960 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.683964 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.683969 | controller |     - '        '
2026-03-14 09:56:26.683973 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.683977 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.683981 | controller |     - '        '
2026-03-14 09:56:26.683986 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.683990 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.683994 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.683998 | controller |     - '          exit 1'
2026-03-14 09:56:26.684003 | controller |     - '        fi'
2026-03-14 09:56:26.684007 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.684011 | controller |     - '        '
2026-03-14 09:56:26.684015 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.684020 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.684024 | controller |       | base64 -d)'
2026-03-14 09:56:26.684028 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.684032 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.684037 | controller |     - '          exit 1'
2026-03-14 09:56:26.684041 | controller |     - '        fi'
2026-03-14 09:56:26.684045 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.684049 | controller |     - '        '
2026-03-14 09:56:26.684053 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.684058 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.684062 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.684076 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.684080 | controller |       --timeout=300s'
2026-03-14 09:56:26.684084 | controller |     - '        '
2026-03-14 09:56:26.684088 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.684093 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.684097 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.684101 | controller |     - '        '
2026-03-14 09:56:26.684105 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.684110 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.684114 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.684118 | controller |     - '        '
2026-03-14 09:56:26.684122 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.684127 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.684134 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.684138 | controller |       available"'
2026-03-14 09:56:26.684142 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.684147 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.684157 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.684162 | controller |     - '          exit 1'
2026-03-14 09:56:26.684166 | controller |     - '        fi'
2026-03-14 09:56:26.684170 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.684175 | controller |     - '        '
2026-03-14 09:56:26.684179 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.684183 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.684188 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.684192 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.684196 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.684200 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.684205 | controller |       -- \'
2026-03-14 09:56:26.684209 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.684213 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.684218 | controller |     - '    logger.go:42: 09:40:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.684222 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.684226 | controller |     - '    logger.go:42: 09:40:36 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.684230 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.684235 | controller |       found'
2026-03-14 09:56:26.684242 | controller |     - '    logger.go:42: 09:40:37 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.684246 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.684250 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.684255 | controller |     - '        '
2026-03-14 09:56:26.684259 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.684263 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.684267 | controller |     - '        '
2026-03-14 09:56:26.684272 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.684276 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.684280 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.684284 | controller |     - '          exit 1'
2026-03-14 09:56:26.684289 | controller |     - '        fi'
2026-03-14 09:56:26.684293 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.684297 | controller |     - '        '
2026-03-14 09:56:26.684301 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.684306 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.684310 | controller |       | base64 -d)'
2026-03-14 09:56:26.684314 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.684318 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.684322 | controller |     - '          exit 1'
2026-03-14 09:56:26.684327 | controller |     - '        fi'
2026-03-14 09:56:26.684331 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.684337 | controller |     - '        '
2026-03-14 09:56:26.684342 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.684346 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.684350 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.684354 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.684359 | controller |       --timeout=300s'
2026-03-14 09:56:26.684365 | controller |     - '        '
2026-03-14 09:56:26.684369 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.684374 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.684378 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.684384 | controller |     - '        '
2026-03-14 09:56:26.684389 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.684393 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.684398 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.684403 | controller |     - '        '
2026-03-14 09:56:26.684408 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.684413 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.684417 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.684421 | controller |       available"'
2026-03-14 09:56:26.684425 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.684429 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.684434 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.684438 | controller |     - '          exit 1'
2026-03-14 09:56:26.684449 | controller |     - '        fi'
2026-03-14 09:56:26.684454 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.684458 | controller |     - '        '
2026-03-14 09:56:26.684462 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.684466 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.684471 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.684475 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.684479 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.684483 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.684488 | controller |       -- \'
2026-03-14 09:56:26.684492 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.684496 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.684500 | controller |     - '    logger.go:42: 09:40:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.684505 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.684509 | controller |     - '    logger.go:42: 09:40:38 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.684513 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.684517 | controller |       found'
2026-03-14 09:56:26.684525 | controller |     - '    logger.go:42: 09:40:39 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.684529 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.684533 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.684538 | controller |     - '        '
2026-03-14 09:56:26.684542 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.684546 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.684550 | controller |     - '        '
2026-03-14 09:56:26.684555 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.684559 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.684563 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.684567 | controller |     - '          exit 1'
2026-03-14 09:56:26.684571 | controller |     - '        fi'
2026-03-14 09:56:26.684576 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.684580 | controller |     - '        '
2026-03-14 09:56:26.684584 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.684588 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.684593 | controller |       | base64 -d)'
2026-03-14 09:56:26.684609 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.684614 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.684618 | controller |     - '          exit 1'
2026-03-14 09:56:26.684622 | controller |     - '        fi'
2026-03-14 09:56:26.684636 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.684640 | controller |     - '        '
2026-03-14 09:56:26.684644 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.684648 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.684653 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.684657 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.684663 | controller |       --timeout=300s'
2026-03-14 09:56:26.684668 | controller |     - '        '
2026-03-14 09:56:26.684672 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.684676 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.684680 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.684685 | controller |     - '        '
2026-03-14 09:56:26.684689 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.684693 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.684697 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.684702 | controller |     - '        '
2026-03-14 09:56:26.684706 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.684710 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.684714 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.684718 | controller |       available"'
2026-03-14 09:56:26.684723 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.684727 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.684733 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.684737 | controller |     - '          exit 1'
2026-03-14 09:56:26.684743 | controller |     - '        fi'
2026-03-14 09:56:26.684755 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.684760 | controller |     - '        '
2026-03-14 09:56:26.684765 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.684771 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.684777 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.684785 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.684790 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.684794 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.684798 | controller |       -- \'
2026-03-14 09:56:26.684802 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.684807 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.684811 | controller |     - '    logger.go:42: 09:40:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.684815 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.684819 | controller |     - '    logger.go:42: 09:40:39 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.684823 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.684828 | controller |       found'
2026-03-14 09:56:26.684832 | controller |     - '    logger.go:42: 09:40:40 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.684836 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.684840 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.684845 | controller |     - '        '
2026-03-14 09:56:26.684849 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.684853 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.684857 | controller |     - '        '
2026-03-14 09:56:26.684862 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.684866 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.684870 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.684874 | controller |     - '          exit 1'
2026-03-14 09:56:26.684879 | controller |     - '        fi'
2026-03-14 09:56:26.684883 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.684887 | controller |     - '        '
2026-03-14 09:56:26.684891 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.684896 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.684900 | controller |       | base64 -d)'
2026-03-14 09:56:26.684904 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.684908 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.684913 | controller |     - '          exit 1'
2026-03-14 09:56:26.684917 | controller |     - '        fi'
2026-03-14 09:56:26.684921 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.684925 | controller |     - '        '
2026-03-14 09:56:26.684930 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.684934 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.684938 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.684942 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.684947 | controller |       --timeout=300s'
2026-03-14 09:56:26.684953 | controller |     - '        '
2026-03-14 09:56:26.684957 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.684961 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.684966 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.684970 | controller |     - '        '
2026-03-14 09:56:26.684974 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.684978 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.684983 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.684987 | controller |     - '        '
2026-03-14 09:56:26.684991 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.684995 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.685000 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.685004 | controller |       available"'
2026-03-14 09:56:26.685008 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.685012 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.685016 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.685021 | controller |     - '          exit 1'
2026-03-14 09:56:26.685025 | controller |     - '        fi'
2026-03-14 09:56:26.685029 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.685040 | controller |     - '        '
2026-03-14 09:56:26.685045 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.685049 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.685053 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.685058 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.685062 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.685066 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.685070 | controller |       -- \'
2026-03-14 09:56:26.685075 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.685079 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.685083 | controller |     - '    logger.go:42: 09:40:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.685090 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.685094 | controller |     - '    logger.go:42: 09:40:40 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.685098 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.685103 | controller |       found'
2026-03-14 09:56:26.685107 | controller |     - '    logger.go:42: 09:40:41 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.685111 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.685115 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.685120 | controller |     - '        '
2026-03-14 09:56:26.685124 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.685128 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.685132 | controller |     - '        '
2026-03-14 09:56:26.685137 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.685142 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.685147 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.685151 | controller |     - '          exit 1'
2026-03-14 09:56:26.685155 | controller |     - '        fi'
2026-03-14 09:56:26.685160 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.685166 | controller |     - '        '
2026-03-14 09:56:26.685170 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.685174 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.685181 | controller |       | base64 -d)'
2026-03-14 09:56:26.685185 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.685189 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.685193 | controller |     - '          exit 1'
2026-03-14 09:56:26.685198 | controller |     - '        fi'
2026-03-14 09:56:26.685202 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.685206 | controller |     - '        '
2026-03-14 09:56:26.685211 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.685215 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.685219 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.685223 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.685227 | controller |       --timeout=300s'
2026-03-14 09:56:26.685232 | controller |     - '        '
2026-03-14 09:56:26.685236 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.685240 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.685245 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.685249 | controller |     - '        '
2026-03-14 09:56:26.685253 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.685257 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.685262 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.685266 | controller |     - '        '
2026-03-14 09:56:26.685270 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.685280 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.685285 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.685289 | controller |       available"'
2026-03-14 09:56:26.685293 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.685297 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.685302 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.685306 | controller |     - '          exit 1'
2026-03-14 09:56:26.685310 | controller |     - '        fi'
2026-03-14 09:56:26.685314 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.685319 | controller |     - '        '
2026-03-14 09:56:26.685330 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.685334 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.685338 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.685343 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.685349 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.685353 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.685358 | controller |       -- \'
2026-03-14 09:56:26.685362 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.685366 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.685370 | controller |     - '    logger.go:42: 09:40:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.685375 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.685379 | controller |     - '    logger.go:42: 09:40:41 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.685383 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.685387 | controller |       found'
2026-03-14 09:56:26.685392 | controller |     - '    logger.go:42: 09:40:42 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.685396 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.685400 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.685404 | controller |     - '        '
2026-03-14 09:56:26.685409 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.685413 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.685417 | controller |     - '        '
2026-03-14 09:56:26.685422 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.685426 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.685432 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.685437 | controller |     - '          exit 1'
2026-03-14 09:56:26.685441 | controller |     - '        fi'
2026-03-14 09:56:26.685445 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.685450 | controller |     - '        '
2026-03-14 09:56:26.685456 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.685460 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.685465 | controller |       | base64 -d)'
2026-03-14 09:56:26.685469 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.685473 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.685477 | controller |     - '          exit 1'
2026-03-14 09:56:26.685482 | controller |     - '        fi'
2026-03-14 09:56:26.685486 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.685490 | controller |     - '        '
2026-03-14 09:56:26.685495 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.685499 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.685503 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.685508 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.685512 | controller |       --timeout=300s'
2026-03-14 09:56:26.685516 | controller |     - '        '
2026-03-14 09:56:26.685520 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.685525 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.685529 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.685533 | controller |     - '        '
2026-03-14 09:56:26.685538 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.685544 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.685548 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.685553 | controller |     - '        '
2026-03-14 09:56:26.685557 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.685561 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.685565 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.685570 | controller |       available"'
2026-03-14 09:56:26.685574 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.685578 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.685583 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.685587 | controller |     - '          exit 1'
2026-03-14 09:56:26.685591 | controller |     - '        fi'
2026-03-14 09:56:26.685608 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.685613 | controller |     - '        '
2026-03-14 09:56:26.685618 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.685628 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.685633 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.685637 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.685642 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.685646 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.685650 | controller |       -- \'
2026-03-14 09:56:26.685655 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.685659 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.685664 | controller |     - '    logger.go:42: 09:40:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.685668 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.685672 | controller |     - '    logger.go:42: 09:40:42 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.685677 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.685681 | controller |       found'
2026-03-14 09:56:26.685685 | controller |     - '    logger.go:42: 09:40:43 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.685690 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.685694 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.685699 | controller |     - '        '
2026-03-14 09:56:26.685703 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.685707 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.685711 | controller |     - '        '
2026-03-14 09:56:26.685716 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.685720 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.685724 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.685728 | controller |     - '          exit 1'
2026-03-14 09:56:26.685732 | controller |     - '        fi'
2026-03-14 09:56:26.685737 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.685741 | controller |     - '        '
2026-03-14 09:56:26.685745 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.685753 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.685757 | controller |       | base64 -d)'
2026-03-14 09:56:26.685761 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.685766 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.685770 | controller |     - '          exit 1'
2026-03-14 09:56:26.685774 | controller |     - '        fi'
2026-03-14 09:56:26.685778 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.685783 | controller |     - '        '
2026-03-14 09:56:26.685787 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.685791 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.685795 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.685800 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.685804 | controller |       --timeout=300s'
2026-03-14 09:56:26.685808 | controller |     - '        '
2026-03-14 09:56:26.685812 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.685817 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.685821 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.685829 | controller |     - '        '
2026-03-14 09:56:26.685835 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.685841 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.685847 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.685853 | controller |     - '        '
2026-03-14 09:56:26.685859 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.685865 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.685871 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.685879 | controller |       available"'
2026-03-14 09:56:26.685884 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.685888 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.685892 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.685896 | controller |     - '          exit 1'
2026-03-14 09:56:26.685901 | controller |     - '        fi'
2026-03-14 09:56:26.685905 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.685909 | controller |     - '        '
2026-03-14 09:56:26.685914 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.685928 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.685932 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.685937 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.685941 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.685945 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.685950 | controller |       -- \'
2026-03-14 09:56:26.685954 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.685958 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.685965 | controller |     - '    logger.go:42: 09:40:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.685969 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.685973 | controller |     - '    logger.go:42: 09:40:43 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.685978 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.685982 | controller |       found'
2026-03-14 09:56:26.685986 | controller |     - '    logger.go:42: 09:40:44 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.685990 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.685995 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.685999 | controller |     - '        '
2026-03-14 09:56:26.686003 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.686008 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.686012 | controller |     - '        '
2026-03-14 09:56:26.686016 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.686020 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.686025 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.686029 | controller |     - '          exit 1'
2026-03-14 09:56:26.686033 | controller |     - '        fi'
2026-03-14 09:56:26.686038 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.686042 | controller |     - '        '
2026-03-14 09:56:26.686046 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.686052 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.686057 | controller |       | base64 -d)'
2026-03-14 09:56:26.686061 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.686065 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.686069 | controller |     - '          exit 1'
2026-03-14 09:56:26.686074 | controller |     - '        fi'
2026-03-14 09:56:26.686078 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.686082 | controller |     - '        '
2026-03-14 09:56:26.686087 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.686091 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.686095 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.686099 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.686104 | controller |       --timeout=300s'
2026-03-14 09:56:26.686108 | controller |     - '        '
2026-03-14 09:56:26.686112 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.686116 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.686121 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.686125 | controller |     - '        '
2026-03-14 09:56:26.686129 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.686133 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.686138 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.686142 | controller |     - '        '
2026-03-14 09:56:26.686146 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.686150 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.686156 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.686161 | controller |       available"'
2026-03-14 09:56:26.686165 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.686169 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.686173 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.686178 | controller |     - '          exit 1'
2026-03-14 09:56:26.686182 | controller |     - '        fi'
2026-03-14 09:56:26.686186 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.686190 | controller |     - '        '
2026-03-14 09:56:26.686195 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.686199 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.686210 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.686214 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.686218 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.686223 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.686227 | controller |       -- \'
2026-03-14 09:56:26.686231 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.686236 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.686240 | controller |     - '    logger.go:42: 09:40:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.686244 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.686249 | controller |     - '    logger.go:42: 09:40:45 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.686253 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.686257 | controller |       found'
2026-03-14 09:56:26.686261 | controller |     - '    logger.go:42: 09:40:46 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.686266 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.686270 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.686274 | controller |     - '        '
2026-03-14 09:56:26.686278 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.686283 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.686287 | controller |     - '        '
2026-03-14 09:56:26.686291 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.686296 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.686300 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.686304 | controller |     - '          exit 1'
2026-03-14 09:56:26.686308 | controller |     - '        fi'
2026-03-14 09:56:26.686313 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.686317 | controller |     - '        '
2026-03-14 09:56:26.686321 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.686325 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.686330 | controller |       | base64 -d)'
2026-03-14 09:56:26.686334 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.686338 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.686343 | controller |     - '          exit 1'
2026-03-14 09:56:26.686347 | controller |     - '        fi'
2026-03-14 09:56:26.686351 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.686357 | controller |     - '        '
2026-03-14 09:56:26.686361 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.686366 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.686370 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.686374 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.686378 | controller |       --timeout=300s'
2026-03-14 09:56:26.686383 | controller |     - '        '
2026-03-14 09:56:26.686387 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.686391 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.686396 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.686400 | controller |     - '        '
2026-03-14 09:56:26.686404 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.686408 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.686413 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.686417 | controller |     - '        '
2026-03-14 09:56:26.686421 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.686426 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.686430 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.686434 | controller |       available"'
2026-03-14 09:56:26.686438 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.686443 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.686447 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.686451 | controller |     - '          exit 1'
2026-03-14 09:56:26.686456 | controller |     - '        fi'
2026-03-14 09:56:26.686460 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.686466 | controller |     - '        '
2026-03-14 09:56:26.686472 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.686476 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.686486 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.686491 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.686495 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.686500 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.686506 | controller |       -- \'
2026-03-14 09:56:26.686510 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.686514 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.686518 | controller |     - '    logger.go:42: 09:40:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.686523 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.686527 | controller |     - '    logger.go:42: 09:40:46 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.686531 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.686535 | controller |       found'
2026-03-14 09:56:26.686540 | controller |     - '    logger.go:42: 09:40:47 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.686546 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.686550 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.686554 | controller |     - '        '
2026-03-14 09:56:26.686559 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.686563 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.686567 | controller |     - '        '
2026-03-14 09:56:26.686571 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.686575 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.686580 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.686584 | controller |     - '          exit 1'
2026-03-14 09:56:26.686588 | controller |     - '        fi'
2026-03-14 09:56:26.686593 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.686610 | controller |     - '        '
2026-03-14 09:56:26.686614 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.686619 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.686623 | controller |       | base64 -d)'
2026-03-14 09:56:26.686627 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.686632 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.686636 | controller |     - '          exit 1'
2026-03-14 09:56:26.686640 | controller |     - '        fi'
2026-03-14 09:56:26.686644 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.686649 | controller |     - '        '
2026-03-14 09:56:26.686653 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.686657 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.686661 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.686666 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.686670 | controller |       --timeout=300s'
2026-03-14 09:56:26.686674 | controller |     - '        '
2026-03-14 09:56:26.686678 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.686683 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.686687 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.686691 | controller |     - '        '
2026-03-14 09:56:26.686695 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.686700 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.686704 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.686708 | controller |     - '        '
2026-03-14 09:56:26.686713 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.686717 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.686721 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.686725 | controller |       available"'
2026-03-14 09:56:26.686730 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.686734 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.686738 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.686742 | controller |     - '          exit 1'
2026-03-14 09:56:26.686747 | controller |     - '        fi'
2026-03-14 09:56:26.686751 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.686758 | controller |     - '        '
2026-03-14 09:56:26.686763 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.686767 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.686771 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.686782 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.686787 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.686794 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.686798 | controller |       -- \'
2026-03-14 09:56:26.686803 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.686807 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.686811 | controller |     - '    logger.go:42: 09:40:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.686816 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.686820 | controller |     - '    logger.go:42: 09:40:47 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.686824 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.686829 | controller |       found'
2026-03-14 09:56:26.686833 | controller |     - '    logger.go:42: 09:40:48 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.686837 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.686842 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.686846 | controller |     - '        '
2026-03-14 09:56:26.686850 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.686855 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.686859 | controller |     - '        '
2026-03-14 09:56:26.686863 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.686867 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.686872 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.686876 | controller |     - '          exit 1'
2026-03-14 09:56:26.686880 | controller |     - '        fi'
2026-03-14 09:56:26.686885 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.686889 | controller |     - '        '
2026-03-14 09:56:26.686893 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.686898 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.686902 | controller |       | base64 -d)'
2026-03-14 09:56:26.686906 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.686911 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.686915 | controller |     - '          exit 1'
2026-03-14 09:56:26.686919 | controller |     - '        fi'
2026-03-14 09:56:26.686924 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.686928 | controller |     - '        '
2026-03-14 09:56:26.686932 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.686937 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.686941 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.686946 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.686952 | controller |       --timeout=300s'
2026-03-14 09:56:26.686958 | controller |     - '        '
2026-03-14 09:56:26.686964 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.686971 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.686992 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.686998 | controller |     - '        '
2026-03-14 09:56:26.687004 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.687010 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.687015 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.687021 | controller |     - '        '
2026-03-14 09:56:26.687027 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.687032 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.687038 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.687044 | controller |       available"'
2026-03-14 09:56:26.687049 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.687055 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.687060 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.687066 | controller |     - '          exit 1'
2026-03-14 09:56:26.687072 | controller |     - '        fi'
2026-03-14 09:56:26.687077 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.687083 | controller |     - '        '
2026-03-14 09:56:26.687088 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.687097 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.687103 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.687118 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.687124 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.687129 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.687135 | controller |       -- \'
2026-03-14 09:56:26.687141 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.687146 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.687152 | controller |     - '    logger.go:42: 09:40:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.687157 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.687163 | controller |     - '    logger.go:42: 09:40:48 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.687168 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.687174 | controller |       found'
2026-03-14 09:56:26.687180 | controller |     - '    logger.go:42: 09:40:49 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.687185 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.687191 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.687196 | controller |     - '        '
2026-03-14 09:56:26.687202 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.687207 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.687213 | controller |     - '        '
2026-03-14 09:56:26.687218 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.687229 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.687235 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.687241 | controller |     - '          exit 1'
2026-03-14 09:56:26.687246 | controller |     - '        fi'
2026-03-14 09:56:26.687252 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.687257 | controller |     - '        '
2026-03-14 09:56:26.687263 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.687268 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.687274 | controller |       | base64 -d)'
2026-03-14 09:56:26.687279 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.687285 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.687290 | controller |     - '          exit 1'
2026-03-14 09:56:26.687296 | controller |     - '        fi'
2026-03-14 09:56:26.687300 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.687304 | controller |     - '        '
2026-03-14 09:56:26.687309 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.687313 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.687317 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.687321 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.687326 | controller |       --timeout=300s'
2026-03-14 09:56:26.687330 | controller |     - '        '
2026-03-14 09:56:26.687334 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.687339 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.687343 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.687347 | controller |     - '        '
2026-03-14 09:56:26.687351 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.687356 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.687360 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.687364 | controller |     - '        '
2026-03-14 09:56:26.687368 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.687373 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.687377 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.687381 | controller |       available"'
2026-03-14 09:56:26.687385 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.687390 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.687396 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.687401 | controller |     - '          exit 1'
2026-03-14 09:56:26.687405 | controller |     - '        fi'
2026-03-14 09:56:26.687409 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.687413 | controller |     - '        '
2026-03-14 09:56:26.687418 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.687422 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.687426 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.687430 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.687444 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.687448 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.687452 | controller |       -- \'
2026-03-14 09:56:26.687457 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.687461 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.687465 | controller |     - '    logger.go:42: 09:40:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.687470 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.687474 | controller |     - '    logger.go:42: 09:40:49 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.687478 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.687482 | controller |       found'
2026-03-14 09:56:26.687487 | controller |     - '    logger.go:42: 09:40:50 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.687491 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.687497 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.687502 | controller |     - '        '
2026-03-14 09:56:26.687506 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.687510 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.687514 | controller |     - '        '
2026-03-14 09:56:26.687519 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.687523 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.687527 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.687531 | controller |     - '          exit 1'
2026-03-14 09:56:26.687536 | controller |     - '        fi'
2026-03-14 09:56:26.687540 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.687544 | controller |     - '        '
2026-03-14 09:56:26.687548 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.687553 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.687557 | controller |       | base64 -d)'
2026-03-14 09:56:26.687561 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.687565 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.687570 | controller |     - '          exit 1'
2026-03-14 09:56:26.687574 | controller |     - '        fi'
2026-03-14 09:56:26.687578 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.687582 | controller |     - '        '
2026-03-14 09:56:26.687587 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.687591 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.687606 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.687613 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.687617 | controller |       --timeout=300s'
2026-03-14 09:56:26.687621 | controller |     - '        '
2026-03-14 09:56:26.687626 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.687630 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.687636 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.687641 | controller |     - '        '
2026-03-14 09:56:26.687645 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.687649 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.687656 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.687660 | controller |     - '        '
2026-03-14 09:56:26.687665 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.687669 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.687673 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.687677 | controller |       available"'
2026-03-14 09:56:26.687682 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.687686 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.687690 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.687695 | controller |     - '          exit 1'
2026-03-14 09:56:26.687699 | controller |     - '        fi'
2026-03-14 09:56:26.687703 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.687707 | controller |     - '        '
2026-03-14 09:56:26.687712 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.687716 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.687720 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.687725 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.687737 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.687742 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.687746 | controller |       -- \'
2026-03-14 09:56:26.687751 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.687755 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.687759 | controller |     - '    logger.go:42: 09:40:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.687764 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.687768 | controller |     - '    logger.go:42: 09:40:50 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.687772 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.687776 | controller |       found'
2026-03-14 09:56:26.687781 | controller |     - '    logger.go:42: 09:40:51 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.687785 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.687789 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.687793 | controller |     - '        '
2026-03-14 09:56:26.687798 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.687802 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.687806 | controller |     - '        '
2026-03-14 09:56:26.687810 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.687815 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.687819 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.687823 | controller |     - '          exit 1'
2026-03-14 09:56:26.687827 | controller |     - '        fi'
2026-03-14 09:56:26.687832 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.687836 | controller |     - '        '
2026-03-14 09:56:26.687840 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.687844 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.687850 | controller |       | base64 -d)'
2026-03-14 09:56:26.687855 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.687859 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.687863 | controller |     - '          exit 1'
2026-03-14 09:56:26.687868 | controller |     - '        fi'
2026-03-14 09:56:26.687872 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.687876 | controller |     - '        '
2026-03-14 09:56:26.687881 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.687885 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.687889 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.687893 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.687897 | controller |       --timeout=300s'
2026-03-14 09:56:26.687902 | controller |     - '        '
2026-03-14 09:56:26.687906 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.687912 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.687916 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.687921 | controller |     - '        '
2026-03-14 09:56:26.687925 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.687929 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.687935 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.687939 | controller |     - '        '
2026-03-14 09:56:26.687944 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.687948 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.687952 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.687957 | controller |       available"'
2026-03-14 09:56:26.687961 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.687965 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.687969 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.687974 | controller |     - '          exit 1'
2026-03-14 09:56:26.687978 | controller |     - '        fi'
2026-03-14 09:56:26.687982 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.687986 | controller |     - '        '
2026-03-14 09:56:26.687991 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.687995 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.687999 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.688003 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.688008 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.688018 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.688023 | controller |       -- \'
2026-03-14 09:56:26.688027 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.688031 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.688037 | controller |     - '    logger.go:42: 09:40:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.688043 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.688049 | controller |     - '    logger.go:42: 09:40:52 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.688055 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.688061 | controller |       found'
2026-03-14 09:56:26.688066 | controller |     - '    logger.go:42: 09:40:53 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.688070 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.688075 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.688079 | controller |     - '        '
2026-03-14 09:56:26.688083 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.688088 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.688092 | controller |     - '        '
2026-03-14 09:56:26.688096 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.688100 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.688105 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.688109 | controller |     - '          exit 1'
2026-03-14 09:56:26.688113 | controller |     - '        fi'
2026-03-14 09:56:26.688117 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.688122 | controller |     - '        '
2026-03-14 09:56:26.688126 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.688130 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.688135 | controller |       | base64 -d)'
2026-03-14 09:56:26.688139 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.688143 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.688147 | controller |     - '          exit 1'
2026-03-14 09:56:26.688152 | controller |     - '        fi'
2026-03-14 09:56:26.688156 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.688160 | controller |     - '        '
2026-03-14 09:56:26.688165 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.688169 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.688173 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.688178 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.688182 | controller |       --timeout=300s'
2026-03-14 09:56:26.688186 | controller |     - '        '
2026-03-14 09:56:26.688190 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.688195 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.688199 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.688203 | controller |     - '        '
2026-03-14 09:56:26.688208 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.688212 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.688216 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.688220 | controller |     - '        '
2026-03-14 09:56:26.688225 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.688229 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.688233 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.688239 | controller |       available"'
2026-03-14 09:56:26.688244 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.688248 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.688252 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.688257 | controller |     - '          exit 1'
2026-03-14 09:56:26.688261 | controller |     - '        fi'
2026-03-14 09:56:26.688265 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.688269 | controller |     - '        '
2026-03-14 09:56:26.688274 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.688278 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.688282 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.688286 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.688291 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.688304 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.688309 | controller |       -- \'
2026-03-14 09:56:26.688313 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.688318 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.688322 | controller |     - '    logger.go:42: 09:40:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.688327 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.688331 | controller |     - '    logger.go:42: 09:40:53 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.688336 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.688341 | controller |       found'
2026-03-14 09:56:26.688346 | controller |     - '    logger.go:42: 09:40:54 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.688350 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.688356 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.688361 | controller |     - '        '
2026-03-14 09:56:26.688366 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.688370 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.688374 | controller |     - '        '
2026-03-14 09:56:26.688379 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.688383 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.688387 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.688391 | controller |     - '          exit 1'
2026-03-14 09:56:26.688396 | controller |     - '        fi'
2026-03-14 09:56:26.688400 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.688404 | controller |     - '        '
2026-03-14 09:56:26.688409 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.688413 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.688417 | controller |       | base64 -d)'
2026-03-14 09:56:26.688421 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.688426 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.688430 | controller |     - '          exit 1'
2026-03-14 09:56:26.688434 | controller |     - '        fi'
2026-03-14 09:56:26.688439 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.688443 | controller |     - '        '
2026-03-14 09:56:26.688447 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.688453 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.688457 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.688462 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.688466 | controller |       --timeout=300s'
2026-03-14 09:56:26.688470 | controller |     - '        '
2026-03-14 09:56:26.688475 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.688479 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.688483 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.688487 | controller |     - '        '
2026-03-14 09:56:26.688492 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.688496 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.688500 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.688505 | controller |     - '        '
2026-03-14 09:56:26.688509 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.688513 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.688517 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.688522 | controller |       available"'
2026-03-14 09:56:26.688526 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.688530 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.688535 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.688539 | controller |     - '          exit 1'
2026-03-14 09:56:26.688543 | controller |     - '        fi'
2026-03-14 09:56:26.688547 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.688552 | controller |     - '        '
2026-03-14 09:56:26.688556 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.688560 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.688565 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.688569 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.688573 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.688577 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.688582 | controller |       -- \'
2026-03-14 09:56:26.688592 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.688610 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.688614 | controller |     - '    logger.go:42: 09:40:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.688618 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.688623 | controller |     - '    logger.go:42: 09:40:54 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.688627 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.688631 | controller |       found'
2026-03-14 09:56:26.688635 | controller |     - '    logger.go:42: 09:40:55 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.688640 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.688647 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.688651 | controller |     - '        '
2026-03-14 09:56:26.688655 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.688660 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.688664 | controller |     - '        '
2026-03-14 09:56:26.688668 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.688672 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.688677 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.688681 | controller |     - '          exit 1'
2026-03-14 09:56:26.688685 | controller |     - '        fi'
2026-03-14 09:56:26.688689 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.688694 | controller |     - '        '
2026-03-14 09:56:26.688698 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.688706 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.688710 | controller |       | base64 -d)'
2026-03-14 09:56:26.688714 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.688719 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.688723 | controller |     - '          exit 1'
2026-03-14 09:56:26.688729 | controller |     - '        fi'
2026-03-14 09:56:26.688733 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.688738 | controller |     - '        '
2026-03-14 09:56:26.688742 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.688746 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.688750 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.688755 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.688759 | controller |       --timeout=300s'
2026-03-14 09:56:26.688763 | controller |     - '        '
2026-03-14 09:56:26.688767 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.688772 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.688776 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.688780 | controller |     - '        '
2026-03-14 09:56:26.688784 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.688789 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.688793 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.688797 | controller |     - '        '
2026-03-14 09:56:26.688801 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.688806 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.688810 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.688814 | controller |       available"'
2026-03-14 09:56:26.688818 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.688823 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.688827 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.688831 | controller |     - '          exit 1'
2026-03-14 09:56:26.688835 | controller |     - '        fi'
2026-03-14 09:56:26.688840 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.688845 | controller |     - '        '
2026-03-14 09:56:26.688850 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.688854 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.688858 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.688862 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.688867 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.688871 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.688875 | controller |       -- \'
2026-03-14 09:56:26.688887 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.688893 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.688897 | controller |     - '    logger.go:42: 09:40:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.688902 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.688906 | controller |     - '    logger.go:42: 09:40:55 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.688912 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.688916 | controller |       found'
2026-03-14 09:56:26.688920 | controller |     - '    logger.go:42: 09:40:56 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.688925 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.688929 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.688933 | controller |     - '        '
2026-03-14 09:56:26.688937 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.688942 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.688946 | controller |     - '        '
2026-03-14 09:56:26.688950 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.688954 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.688959 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.688963 | controller |     - '          exit 1'
2026-03-14 09:56:26.688967 | controller |     - '        fi'
2026-03-14 09:56:26.688971 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.688976 | controller |     - '        '
2026-03-14 09:56:26.688980 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.688984 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.688988 | controller |       | base64 -d)'
2026-03-14 09:56:26.688993 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.688997 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.689001 | controller |     - '          exit 1'
2026-03-14 09:56:26.689005 | controller |     - '        fi'
2026-03-14 09:56:26.689010 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.689014 | controller |     - '        '
2026-03-14 09:56:26.689018 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.689022 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.689027 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.689031 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.689035 | controller |       --timeout=300s'
2026-03-14 09:56:26.689039 | controller |     - '        '
2026-03-14 09:56:26.689044 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.689050 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.689054 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.689058 | controller |     - '        '
2026-03-14 09:56:26.689063 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.689067 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.689071 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.689075 | controller |     - '        '
2026-03-14 09:56:26.689080 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.689084 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.689088 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.689092 | controller |       available"'
2026-03-14 09:56:26.689097 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.689101 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.689105 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.689110 | controller |     - '          exit 1'
2026-03-14 09:56:26.689117 | controller |     - '        fi'
2026-03-14 09:56:26.689123 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.689129 | controller |     - '        '
2026-03-14 09:56:26.689133 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.689137 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.689141 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.689146 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.689150 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.689154 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.689159 | controller |       -- \'
2026-03-14 09:56:26.689163 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.689174 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.689179 | controller |     - '    logger.go:42: 09:40:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.689183 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.689187 | controller |     - '    logger.go:42: 09:40:56 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.689192 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.689196 | controller |       found'
2026-03-14 09:56:26.689200 | controller |     - '    logger.go:42: 09:40:57 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.689204 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.689211 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.689216 | controller |     - '        '
2026-03-14 09:56:26.689220 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.689224 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.689229 | controller |     - '        '
2026-03-14 09:56:26.689233 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.689237 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.689245 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.689249 | controller |     - '          exit 1'
2026-03-14 09:56:26.689253 | controller |     - '        fi'
2026-03-14 09:56:26.689258 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.689264 | controller |     - '        '
2026-03-14 09:56:26.689268 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.689272 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.689278 | controller |       | base64 -d)'
2026-03-14 09:56:26.689282 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.689286 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.689291 | controller |     - '          exit 1'
2026-03-14 09:56:26.689295 | controller |     - '        fi'
2026-03-14 09:56:26.689299 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.689303 | controller |     - '        '
2026-03-14 09:56:26.689307 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.689312 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.689316 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.689320 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.689324 | controller |       --timeout=300s'
2026-03-14 09:56:26.689329 | controller |     - '        '
2026-03-14 09:56:26.689333 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.689337 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.689341 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.689346 | controller |     - '        '
2026-03-14 09:56:26.689350 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.689354 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.689358 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.689363 | controller |     - '        '
2026-03-14 09:56:26.689367 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.689371 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.689375 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.689380 | controller |       available"'
2026-03-14 09:56:26.689384 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.689388 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.689393 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.689397 | controller |     - '          exit 1'
2026-03-14 09:56:26.689401 | controller |     - '        fi'
2026-03-14 09:56:26.689405 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.689410 | controller |     - '        '
2026-03-14 09:56:26.689414 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.689418 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.689422 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.689427 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.689431 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.689437 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.689442 | controller |       -- \'
2026-03-14 09:56:26.689446 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.689457 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.689461 | controller |     - '    logger.go:42: 09:40:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.689465 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.689470 | controller |     - '    logger.go:42: 09:40:58 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.689474 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.689480 | controller |       found'
2026-03-14 09:56:26.689484 | controller |     - '    logger.go:42: 09:40:59 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.689489 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.689493 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.689497 | controller |     - '        '
2026-03-14 09:56:26.689501 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.689506 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.689510 | controller |     - '        '
2026-03-14 09:56:26.689514 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.689518 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.689523 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.689527 | controller |     - '          exit 1'
2026-03-14 09:56:26.689531 | controller |     - '        fi'
2026-03-14 09:56:26.689535 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.689540 | controller |     - '        '
2026-03-14 09:56:26.689544 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.689548 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.689553 | controller |       | base64 -d)'
2026-03-14 09:56:26.689557 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.689561 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.689565 | controller |     - '          exit 1'
2026-03-14 09:56:26.689570 | controller |     - '        fi'
2026-03-14 09:56:26.689574 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.689578 | controller |     - '        '
2026-03-14 09:56:26.689582 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.689587 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.689591 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.689607 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.689613 | controller |       --timeout=300s'
2026-03-14 09:56:26.689617 | controller |     - '        '
2026-03-14 09:56:26.689622 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.689626 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.689630 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.689634 | controller |     - '        '
2026-03-14 09:56:26.689639 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.689643 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.689649 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.689653 | controller |     - '        '
2026-03-14 09:56:26.689658 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.689662 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.689666 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.689670 | controller |       available"'
2026-03-14 09:56:26.689675 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.689679 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.689683 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.689687 | controller |     - '          exit 1'
2026-03-14 09:56:26.689692 | controller |     - '        fi'
2026-03-14 09:56:26.689696 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.689700 | controller |     - '        '
2026-03-14 09:56:26.689704 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.689709 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.689713 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.689717 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.689722 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.689726 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.689730 | controller |       -- \'
2026-03-14 09:56:26.689734 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.689739 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.689750 | controller |     - '    logger.go:42: 09:40:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.689754 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.689759 | controller |     - '    logger.go:42: 09:40:59 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.689763 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.689767 | controller |       found'
2026-03-14 09:56:26.689772 | controller |     - '    logger.go:42: 09:41:00 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.689776 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.689780 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.689785 | controller |     - '        '
2026-03-14 09:56:26.689789 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.689793 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.689798 | controller |     - '        '
2026-03-14 09:56:26.689804 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.689808 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.689812 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.689817 | controller |     - '          exit 1'
2026-03-14 09:56:26.689821 | controller |     - '        fi'
2026-03-14 09:56:26.689825 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.689830 | controller |     - '        '
2026-03-14 09:56:26.689834 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.689838 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.689842 | controller |       | base64 -d)'
2026-03-14 09:56:26.689856 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.689863 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.689867 | controller |     - '          exit 1'
2026-03-14 09:56:26.689872 | controller |     - '        fi'
2026-03-14 09:56:26.689876 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.689880 | controller |     - '        '
2026-03-14 09:56:26.689884 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.689889 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.689893 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.689897 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.689901 | controller |       --timeout=300s'
2026-03-14 09:56:26.689906 | controller |     - '        '
2026-03-14 09:56:26.689910 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.689914 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.689919 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.689923 | controller |     - '        '
2026-03-14 09:56:26.689927 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.689931 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.689936 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.689942 | controller |     - '        '
2026-03-14 09:56:26.689946 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.689950 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.689954 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.689959 | controller |       available"'
2026-03-14 09:56:26.689963 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.689967 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.689973 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.689977 | controller |     - '          exit 1'
2026-03-14 09:56:26.689982 | controller |     - '        fi'
2026-03-14 09:56:26.689986 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.689990 | controller |     - '        '
2026-03-14 09:56:26.689994 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.689999 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.690003 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.690007 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.690012 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.690016 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.690020 | controller |       -- \'
2026-03-14 09:56:26.690024 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.690029 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.690041 | controller |     - '    logger.go:42: 09:41:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.690047 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.690052 | controller |     - '    logger.go:42: 09:41:00 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.690056 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.690060 | controller |       found'
2026-03-14 09:56:26.690065 | controller |     - '    logger.go:42: 09:41:01 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.690071 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.690075 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.690080 | controller |     - '        '
2026-03-14 09:56:26.690084 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.690088 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.690092 | controller |     - '        '
2026-03-14 09:56:26.690097 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.690101 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.690105 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.690109 | controller |     - '          exit 1'
2026-03-14 09:56:26.690114 | controller |     - '        fi'
2026-03-14 09:56:26.690118 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.690122 | controller |     - '        '
2026-03-14 09:56:26.690126 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.690131 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.690135 | controller |       | base64 -d)'
2026-03-14 09:56:26.690139 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.690143 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.690147 | controller |     - '          exit 1'
2026-03-14 09:56:26.690152 | controller |     - '        fi'
2026-03-14 09:56:26.690156 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.690160 | controller |     - '        '
2026-03-14 09:56:26.690164 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.690169 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.690173 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.690179 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.690185 | controller |       --timeout=300s'
2026-03-14 09:56:26.690192 | controller |     - '        '
2026-03-14 09:56:26.690198 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.690203 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.690210 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.690215 | controller |     - '        '
2026-03-14 09:56:26.690219 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.690223 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.690228 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.690232 | controller |     - '        '
2026-03-14 09:56:26.690236 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.690241 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.690245 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.690249 | controller |       available"'
2026-03-14 09:56:26.690255 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.690260 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.690264 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.690268 | controller |     - '          exit 1'
2026-03-14 09:56:26.690273 | controller |     - '        fi'
2026-03-14 09:56:26.690277 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.690281 | controller |     - '        '
2026-03-14 09:56:26.690285 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.690289 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.690294 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.690298 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.690302 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.690306 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.690311 | controller |       -- \'
2026-03-14 09:56:26.690315 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.690319 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.690323 | controller |     - '    logger.go:42: 09:41:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.690336 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.690340 | controller |     - '    logger.go:42: 09:41:01 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.690344 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.690349 | controller |       found'
2026-03-14 09:56:26.690353 | controller |     - '    logger.go:42: 09:41:02 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.690357 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.690361 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.690366 | controller |     - '        '
2026-03-14 09:56:26.690370 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.690374 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.690378 | controller |     - '        '
2026-03-14 09:56:26.690382 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.690387 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.690391 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.690395 | controller |     - '          exit 1'
2026-03-14 09:56:26.690399 | controller |     - '        fi'
2026-03-14 09:56:26.690404 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.690408 | controller |     - '        '
2026-03-14 09:56:26.690412 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.690416 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.690420 | controller |       | base64 -d)'
2026-03-14 09:56:26.690425 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.690429 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.690433 | controller |     - '          exit 1'
2026-03-14 09:56:26.690437 | controller |     - '        fi'
2026-03-14 09:56:26.690441 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.690446 | controller |     - '        '
2026-03-14 09:56:26.690450 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.690454 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.690460 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.690464 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.690468 | controller |       --timeout=300s'
2026-03-14 09:56:26.690473 | controller |     - '        '
2026-03-14 09:56:26.690477 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.690481 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.690485 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.690490 | controller |     - '        '
2026-03-14 09:56:26.690494 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.690498 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.690502 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.690507 | controller |     - '        '
2026-03-14 09:56:26.690511 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.690515 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.690519 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.690524 | controller |       available"'
2026-03-14 09:56:26.690528 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.690532 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.690536 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.690541 | controller |     - '          exit 1'
2026-03-14 09:56:26.690545 | controller |     - '        fi'
2026-03-14 09:56:26.690549 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.690553 | controller |     - '        '
2026-03-14 09:56:26.690558 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.690562 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.690566 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.690570 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.690574 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.690579 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.690585 | controller |       -- \'
2026-03-14 09:56:26.690589 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.690604 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.690611 | controller |     - '    logger.go:42: 09:41:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.690615 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.690626 | controller |     - '    logger.go:42: 09:41:02 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.690630 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.690634 | controller |       found'
2026-03-14 09:56:26.690639 | controller |     - '    logger.go:42: 09:41:03 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.690643 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.690647 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.690654 | controller |     - '        '
2026-03-14 09:56:26.690658 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.690664 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.690668 | controller |     - '        '
2026-03-14 09:56:26.690673 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.690677 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.690681 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.690685 | controller |     - '          exit 1'
2026-03-14 09:56:26.690690 | controller |     - '        fi'
2026-03-14 09:56:26.690694 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.690698 | controller |     - '        '
2026-03-14 09:56:26.690702 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.690707 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.690711 | controller |       | base64 -d)'
2026-03-14 09:56:26.690715 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.690720 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.690724 | controller |     - '          exit 1'
2026-03-14 09:56:26.690728 | controller |     - '        fi'
2026-03-14 09:56:26.690732 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.690736 | controller |     - '        '
2026-03-14 09:56:26.690741 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.690745 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.690749 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.690753 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.690758 | controller |       --timeout=300s'
2026-03-14 09:56:26.690762 | controller |     - '        '
2026-03-14 09:56:26.690766 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.690770 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.690775 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.690779 | controller |     - '        '
2026-03-14 09:56:26.690783 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.690788 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.690792 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.690796 | controller |     - '        '
2026-03-14 09:56:26.690800 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.690805 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.690809 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.690813 | controller |       available"'
2026-03-14 09:56:26.690820 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.690824 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.690828 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.690832 | controller |     - '          exit 1'
2026-03-14 09:56:26.690837 | controller |     - '        fi'
2026-03-14 09:56:26.690841 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.690845 | controller |     - '        '
2026-03-14 09:56:26.690849 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.690855 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.690859 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.690864 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.690868 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.690872 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.690876 | controller |       -- \'
2026-03-14 09:56:26.690881 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.690885 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.690889 | controller |     - '    logger.go:42: 09:41:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.690893 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.690898 | controller |     - '    logger.go:42: 09:41:03 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.690908 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.690913 | controller |       found'
2026-03-14 09:56:26.690917 | controller |     - '    logger.go:42: 09:41:04 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.690921 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.690926 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.690930 | controller |     - '        '
2026-03-14 09:56:26.690934 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.690938 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.690943 | controller |     - '        '
2026-03-14 09:56:26.690947 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.690951 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.690956 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.690960 | controller |     - '          exit 1'
2026-03-14 09:56:26.690964 | controller |     - '        fi'
2026-03-14 09:56:26.690968 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.690973 | controller |     - '        '
2026-03-14 09:56:26.690991 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.690998 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.691004 | controller |       | base64 -d)'
2026-03-14 09:56:26.691010 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.691015 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.691019 | controller |     - '          exit 1'
2026-03-14 09:56:26.691023 | controller |     - '        fi'
2026-03-14 09:56:26.691028 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.691032 | controller |     - '        '
2026-03-14 09:56:26.691036 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.691040 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.691045 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.691049 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.691053 | controller |       --timeout=300s'
2026-03-14 09:56:26.691058 | controller |     - '        '
2026-03-14 09:56:26.691062 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.691066 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.691075 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.691079 | controller |     - '        '
2026-03-14 09:56:26.691083 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.691087 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.691092 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.691096 | controller |     - '        '
2026-03-14 09:56:26.691100 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.691104 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.691109 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.691113 | controller |       available"'
2026-03-14 09:56:26.691117 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.691121 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.691126 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.691130 | controller |     - '          exit 1'
2026-03-14 09:56:26.691134 | controller |     - '        fi'
2026-03-14 09:56:26.691139 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.691143 | controller |     - '        '
2026-03-14 09:56:26.691147 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.691151 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.691156 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.691160 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.691164 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.691169 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.691173 | controller |       -- \'
2026-03-14 09:56:26.691177 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.691181 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.691186 | controller |     - '    logger.go:42: 09:41:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.691190 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.691194 | controller |     - '    logger.go:42: 09:41:05 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.691206 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.691210 | controller |       found'
2026-03-14 09:56:26.691215 | controller |     - '    logger.go:42: 09:41:06 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.691219 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.691223 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.691228 | controller |     - '        '
2026-03-14 09:56:26.691232 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.691236 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.691243 | controller |     - '        '
2026-03-14 09:56:26.691247 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.691252 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.691256 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.691264 | controller |     - '          exit 1'
2026-03-14 09:56:26.691268 | controller |     - '        fi'
2026-03-14 09:56:26.691272 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.691277 | controller |     - '        '
2026-03-14 09:56:26.691281 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.691287 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.691291 | controller |       | base64 -d)'
2026-03-14 09:56:26.691296 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.691300 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.691306 | controller |     - '          exit 1'
2026-03-14 09:56:26.691313 | controller |     - '        fi'
2026-03-14 09:56:26.691327 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.691332 | controller |     - '        '
2026-03-14 09:56:26.691336 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.691340 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.691345 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.691349 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.691353 | controller |       --timeout=300s'
2026-03-14 09:56:26.691357 | controller |     - '        '
2026-03-14 09:56:26.691362 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.691366 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.691370 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.691375 | controller |     - '        '
2026-03-14 09:56:26.691379 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.691383 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.691388 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.691392 | controller |     - '        '
2026-03-14 09:56:26.691396 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.691401 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.691405 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.691409 | controller |       available"'
2026-03-14 09:56:26.691414 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.691418 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.691422 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.691427 | controller |     - '          exit 1'
2026-03-14 09:56:26.691431 | controller |     - '        fi'
2026-03-14 09:56:26.691435 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.691439 | controller |     - '        '
2026-03-14 09:56:26.691444 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.691448 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.691452 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.691457 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.691461 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.691467 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.691472 | controller |       -- \'
2026-03-14 09:56:26.691476 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.691480 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.691485 | controller |     - '    logger.go:42: 09:41:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.691489 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.691493 | controller |     - '    logger.go:42: 09:41:06 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.691498 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.691502 | controller |       found'
2026-03-14 09:56:26.691514 | controller |     - '    logger.go:42: 09:41:07 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.691519 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.691523 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.691528 | controller |     - '        '
2026-03-14 09:56:26.691532 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.691536 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.691541 | controller |     - '        '
2026-03-14 09:56:26.691545 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.691549 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.691554 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.691558 | controller |     - '          exit 1'
2026-03-14 09:56:26.691562 | controller |     - '        fi'
2026-03-14 09:56:26.691566 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.691571 | controller |     - '        '
2026-03-14 09:56:26.691575 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.691579 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.691584 | controller |       | base64 -d)'
2026-03-14 09:56:26.691588 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.691592 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.691610 | controller |     - '          exit 1'
2026-03-14 09:56:26.691614 | controller |     - '        fi'
2026-03-14 09:56:26.691619 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.691623 | controller |     - '        '
2026-03-14 09:56:26.691627 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.691632 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.691636 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.691640 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.691645 | controller |       --timeout=300s'
2026-03-14 09:56:26.691649 | controller |     - '        '
2026-03-14 09:56:26.691654 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.691658 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.691662 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.691667 | controller |     - '        '
2026-03-14 09:56:26.691671 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.691675 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.691679 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.691686 | controller |     - '        '
2026-03-14 09:56:26.691690 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.691695 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.691699 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.691703 | controller |       available"'
2026-03-14 09:56:26.691708 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.691712 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.691716 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.691721 | controller |     - '          exit 1'
2026-03-14 09:56:26.691725 | controller |     - '        fi'
2026-03-14 09:56:26.691729 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.691734 | controller |     - '        '
2026-03-14 09:56:26.691738 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.691742 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.691747 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.691751 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.691761 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.691765 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.691770 | controller |       -- \'
2026-03-14 09:56:26.691774 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.691778 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.691783 | controller |     - '    logger.go:42: 09:41:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.691787 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.691791 | controller |     - '    logger.go:42: 09:41:07 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.691796 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.691800 | controller |       found'
2026-03-14 09:56:26.691811 | controller |     - '    logger.go:42: 09:41:08 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.691815 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.691820 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.691824 | controller |     - '        '
2026-03-14 09:56:26.691829 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.691833 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.691837 | controller |     - '        '
2026-03-14 09:56:26.691842 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.691849 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.691853 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.691858 | controller |     - '          exit 1'
2026-03-14 09:56:26.691862 | controller |     - '        fi'
2026-03-14 09:56:26.691866 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.691871 | controller |     - '        '
2026-03-14 09:56:26.691875 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.691884 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.691889 | controller |       | base64 -d)'
2026-03-14 09:56:26.691893 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.691899 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.691904 | controller |     - '          exit 1'
2026-03-14 09:56:26.691908 | controller |     - '        fi'
2026-03-14 09:56:26.691913 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.691919 | controller |     - '        '
2026-03-14 09:56:26.691923 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.691927 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.691932 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.691936 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.691940 | controller |       --timeout=300s'
2026-03-14 09:56:26.691945 | controller |     - '        '
2026-03-14 09:56:26.691949 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.691955 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.691960 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.691964 | controller |     - '        '
2026-03-14 09:56:26.691968 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.691973 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.691977 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.691981 | controller |     - '        '
2026-03-14 09:56:26.691986 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.691990 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.691994 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.691998 | controller |       available"'
2026-03-14 09:56:26.692003 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.692007 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.692011 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.692015 | controller |     - '          exit 1'
2026-03-14 09:56:26.692020 | controller |     - '        fi'
2026-03-14 09:56:26.692024 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.692028 | controller |     - '        '
2026-03-14 09:56:26.692033 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.692037 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.692041 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.692047 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.692052 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.692056 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.692061 | controller |       -- \'
2026-03-14 09:56:26.692065 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.692069 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.692074 | controller |     - '    logger.go:42: 09:41:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.692080 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.692084 | controller |     - '    logger.go:42: 09:41:08 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.692088 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.692093 | controller |       found'
2026-03-14 09:56:26.692097 | controller |     - '    logger.go:42: 09:41:09 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.692108 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.692112 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.692117 | controller |     - '        '
2026-03-14 09:56:26.692121 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.692125 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.692130 | controller |     - '        '
2026-03-14 09:56:26.692134 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.692139 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.692143 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.692147 | controller |     - '          exit 1'
2026-03-14 09:56:26.692151 | controller |     - '        fi'
2026-03-14 09:56:26.692156 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.692160 | controller |     - '        '
2026-03-14 09:56:26.692164 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.692174 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.692178 | controller |       | base64 -d)'
2026-03-14 09:56:26.692183 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.692187 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.692191 | controller |     - '          exit 1'
2026-03-14 09:56:26.692196 | controller |     - '        fi'
2026-03-14 09:56:26.692200 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.692204 | controller |     - '        '
2026-03-14 09:56:26.692208 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.692213 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.692217 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.692221 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.692225 | controller |       --timeout=300s'
2026-03-14 09:56:26.692230 | controller |     - '        '
2026-03-14 09:56:26.692234 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.692238 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.692243 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.692247 | controller |     - '        '
2026-03-14 09:56:26.692251 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.692257 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.692262 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.692266 | controller |     - '        '
2026-03-14 09:56:26.692271 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.692275 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.692279 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.692284 | controller |       available"'
2026-03-14 09:56:26.692288 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.692294 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.692298 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.692303 | controller |     - '          exit 1'
2026-03-14 09:56:26.692307 | controller |     - '        fi'
2026-03-14 09:56:26.692311 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.692316 | controller |     - '        '
2026-03-14 09:56:26.692320 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.692324 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.692328 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.692333 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.692337 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.692341 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.692346 | controller |       -- \'
2026-03-14 09:56:26.692350 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.692354 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.692359 | controller |     - '    logger.go:42: 09:41:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.692363 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.692367 | controller |     - '    logger.go:42: 09:41:09 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.692372 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.692376 | controller |       found'
2026-03-14 09:56:26.692380 | controller |     - '    logger.go:42: 09:41:10 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.692384 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.692389 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.692400 | controller |     - '        '
2026-03-14 09:56:26.692404 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.692409 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.692413 | controller |     - '        '
2026-03-14 09:56:26.692417 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.692422 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.692426 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.692430 | controller |     - '          exit 1'
2026-03-14 09:56:26.692435 | controller |     - '        fi'
2026-03-14 09:56:26.692439 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.692443 | controller |     - '        '
2026-03-14 09:56:26.692448 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.692452 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.692457 | controller |       | base64 -d)'
2026-03-14 09:56:26.692461 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.692465 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.692470 | controller |     - '          exit 1'
2026-03-14 09:56:26.692474 | controller |     - '        fi'
2026-03-14 09:56:26.692478 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.692483 | controller |     - '        '
2026-03-14 09:56:26.692487 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.692491 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.692499 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.692503 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.692508 | controller |       --timeout=300s'
2026-03-14 09:56:26.692517 | controller |     - '        '
2026-03-14 09:56:26.692522 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.692526 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.692530 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.692535 | controller |     - '        '
2026-03-14 09:56:26.692543 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.692548 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.692552 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.692556 | controller |     - '        '
2026-03-14 09:56:26.692561 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.692565 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.692569 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.692574 | controller |       available"'
2026-03-14 09:56:26.692578 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.692582 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.692589 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.692603 | controller |     - '          exit 1'
2026-03-14 09:56:26.692609 | controller |     - '        fi'
2026-03-14 09:56:26.692614 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.692618 | controller |     - '        '
2026-03-14 09:56:26.692623 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.692627 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.692631 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.692636 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.692640 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.692644 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.692649 | controller |       -- \'
2026-03-14 09:56:26.692653 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.692657 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.692662 | controller |     - '    logger.go:42: 09:41:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.692666 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.692670 | controller |     - '    logger.go:42: 09:41:11 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.692674 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.692679 | controller |       found'
2026-03-14 09:56:26.692683 | controller |     - '    logger.go:42: 09:41:12 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.692687 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.692692 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.692696 | controller |     - '        '
2026-03-14 09:56:26.692710 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.692715 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.692719 | controller |     - '        '
2026-03-14 09:56:26.692724 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.692728 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.692735 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.692739 | controller |     - '          exit 1'
2026-03-14 09:56:26.692744 | controller |     - '        fi'
2026-03-14 09:56:26.692748 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.692752 | controller |     - '        '
2026-03-14 09:56:26.692757 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.692761 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.692765 | controller |       | base64 -d)'
2026-03-14 09:56:26.692769 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.692774 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.692778 | controller |     - '          exit 1'
2026-03-14 09:56:26.692782 | controller |     - '        fi'
2026-03-14 09:56:26.692787 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.692791 | controller |     - '        '
2026-03-14 09:56:26.692795 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.692799 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.692804 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.692808 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.692812 | controller |       --timeout=300s'
2026-03-14 09:56:26.692816 | controller |     - '        '
2026-03-14 09:56:26.692821 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.692825 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.692829 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.692834 | controller |     - '        '
2026-03-14 09:56:26.692838 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.692842 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.692846 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.692851 | controller |     - '        '
2026-03-14 09:56:26.692855 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.692859 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.692864 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.692868 | controller |       available"'
2026-03-14 09:56:26.692872 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.692876 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.692880 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.692885 | controller |     - '          exit 1'
2026-03-14 09:56:26.692889 | controller |     - '        fi'
2026-03-14 09:56:26.692893 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.692897 | controller |     - '        '
2026-03-14 09:56:26.692902 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.692908 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.692912 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.692916 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.692921 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.692925 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.692929 | controller |       -- \'
2026-03-14 09:56:26.692933 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.692938 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.692942 | controller |     - '    logger.go:42: 09:41:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.692946 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.692951 | controller |     - '    logger.go:42: 09:41:12 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.692955 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.692960 | controller |       found'
2026-03-14 09:56:26.692964 | controller |     - '    logger.go:42: 09:41:13 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.692968 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.692973 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.692977 | controller |     - '        '
2026-03-14 09:56:26.692981 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.692997 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.693001 | controller |     - '        '
2026-03-14 09:56:26.693006 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.693010 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.693014 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.693019 | controller |     - '          exit 1'
2026-03-14 09:56:26.693023 | controller |     - '        fi'
2026-03-14 09:56:26.693029 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.693033 | controller |     - '        '
2026-03-14 09:56:26.693038 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.693042 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.693046 | controller |       | base64 -d)'
2026-03-14 09:56:26.693051 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.693055 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.693059 | controller |     - '          exit 1'
2026-03-14 09:56:26.693063 | controller |     - '        fi'
2026-03-14 09:56:26.693068 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.693072 | controller |     - '        '
2026-03-14 09:56:26.693076 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.693080 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.693085 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.693091 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.693095 | controller |       --timeout=300s'
2026-03-14 09:56:26.693099 | controller |     - '        '
2026-03-14 09:56:26.693104 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.693108 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.693114 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.693120 | controller |     - '        '
2026-03-14 09:56:26.693124 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.693128 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.693133 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.693137 | controller |     - '        '
2026-03-14 09:56:26.693141 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.693145 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.693150 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.693154 | controller |       available"'
2026-03-14 09:56:26.693158 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.693162 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.693166 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.693171 | controller |     - '          exit 1'
2026-03-14 09:56:26.693175 | controller |     - '        fi'
2026-03-14 09:56:26.693179 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.693183 | controller |     - '        '
2026-03-14 09:56:26.693188 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.693192 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.693196 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.693200 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.693205 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.693209 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.693213 | controller |       -- \'
2026-03-14 09:56:26.693217 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.693222 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.693226 | controller |     - '    logger.go:42: 09:41:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.693236 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.693240 | controller |     - '    logger.go:42: 09:41:13 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.693244 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.693253 | controller |       found'
2026-03-14 09:56:26.693257 | controller |     - '    logger.go:42: 09:41:14 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.693262 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.693266 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.693270 | controller |     - '        '
2026-03-14 09:56:26.693275 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.693287 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.693292 | controller |     - '        '
2026-03-14 09:56:26.693296 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.693300 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.693305 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.693309 | controller |     - '          exit 1'
2026-03-14 09:56:26.693315 | controller |     - '        fi'
2026-03-14 09:56:26.693320 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.693324 | controller |     - '        '
2026-03-14 09:56:26.693328 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.693333 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.693339 | controller |       | base64 -d)'
2026-03-14 09:56:26.693343 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.693348 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.693352 | controller |     - '          exit 1'
2026-03-14 09:56:26.693356 | controller |     - '        fi'
2026-03-14 09:56:26.693360 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.693365 | controller |     - '        '
2026-03-14 09:56:26.693369 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.693373 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.693377 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.693382 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.693386 | controller |       --timeout=300s'
2026-03-14 09:56:26.693390 | controller |     - '        '
2026-03-14 09:56:26.693394 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.693399 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.693403 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.693407 | controller |     - '        '
2026-03-14 09:56:26.693411 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.693416 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.693420 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.693424 | controller |     - '        '
2026-03-14 09:56:26.693428 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.693433 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.693437 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.693441 | controller |       available"'
2026-03-14 09:56:26.693445 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.693450 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.693454 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.693458 | controller |     - '          exit 1'
2026-03-14 09:56:26.693462 | controller |     - '        fi'
2026-03-14 09:56:26.693467 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.693471 | controller |     - '        '
2026-03-14 09:56:26.693475 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.693479 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.693484 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.693488 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.693492 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.693496 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.693504 | controller |       -- \'
2026-03-14 09:56:26.693510 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.693516 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.693522 | controller |     - '    logger.go:42: 09:41:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.693526 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.693531 | controller |     - '    logger.go:42: 09:41:14 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.693535 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.693539 | controller |       found'
2026-03-14 09:56:26.693543 | controller |     - '    logger.go:42: 09:41:15 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.693548 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.693552 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.693556 | controller |     - '        '
2026-03-14 09:56:26.693560 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.693565 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.693569 | controller |     - '        '
2026-03-14 09:56:26.693580 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.693585 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.693589 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.693604 | controller |     - '          exit 1'
2026-03-14 09:56:26.693611 | controller |     - '        fi'
2026-03-14 09:56:26.693616 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.693620 | controller |     - '        '
2026-03-14 09:56:26.693624 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.693628 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.693633 | controller |       | base64 -d)'
2026-03-14 09:56:26.693637 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.693641 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.693645 | controller |     - '          exit 1'
2026-03-14 09:56:26.693650 | controller |     - '        fi'
2026-03-14 09:56:26.693654 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.693658 | controller |     - '        '
2026-03-14 09:56:26.693662 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.693667 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.693671 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.693675 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.693682 | controller |       --timeout=300s'
2026-03-14 09:56:26.693686 | controller |     - '        '
2026-03-14 09:56:26.693690 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.693695 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.693699 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.693703 | controller |     - '        '
2026-03-14 09:56:26.693707 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.693712 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.693716 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.693723 | controller |     - '        '
2026-03-14 09:56:26.693727 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.693732 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.693736 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.693740 | controller |       available"'
2026-03-14 09:56:26.693744 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.693749 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.693753 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.693757 | controller |     - '          exit 1'
2026-03-14 09:56:26.693761 | controller |     - '        fi'
2026-03-14 09:56:26.693766 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.693770 | controller |     - '        '
2026-03-14 09:56:26.693774 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.693779 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.693783 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.693787 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.693791 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.693796 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.693800 | controller |       -- \'
2026-03-14 09:56:26.693804 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.693808 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.693813 | controller |     - '    logger.go:42: 09:41:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.693817 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.693821 | controller |     - '    logger.go:42: 09:41:15 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.693825 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.693830 | controller |       found'
2026-03-14 09:56:26.693834 | controller |     - '    logger.go:42: 09:41:16 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.693838 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.693842 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.693847 | controller |     - '        '
2026-03-14 09:56:26.693851 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.693855 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.693859 | controller |     - '        '
2026-03-14 09:56:26.693870 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.693875 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.693879 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.693883 | controller |     - '          exit 1'
2026-03-14 09:56:26.693888 | controller |     - '        fi'
2026-03-14 09:56:26.693892 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.693896 | controller |     - '        '
2026-03-14 09:56:26.693900 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.693906 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.693911 | controller |       | base64 -d)'
2026-03-14 09:56:26.693915 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.693919 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.693925 | controller |     - '          exit 1'
2026-03-14 09:56:26.693930 | controller |     - '        fi'
2026-03-14 09:56:26.693934 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.693938 | controller |     - '        '
2026-03-14 09:56:26.693942 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.693947 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.693951 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.693955 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.693959 | controller |       --timeout=300s'
2026-03-14 09:56:26.693964 | controller |     - '        '
2026-03-14 09:56:26.693968 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.693972 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.693982 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.693987 | controller |     - '        '
2026-03-14 09:56:26.693991 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.693995 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.693999 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.694004 | controller |     - '        '
2026-03-14 09:56:26.694008 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.694012 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.694016 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.694021 | controller |       available"'
2026-03-14 09:56:26.694025 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.694029 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.694034 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.694038 | controller |     - '          exit 1'
2026-03-14 09:56:26.694042 | controller |     - '        fi'
2026-03-14 09:56:26.694050 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.694054 | controller |     - '        '
2026-03-14 09:56:26.694058 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.694063 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.694067 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.694071 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.694075 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.694081 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.694085 | controller |       -- \'
2026-03-14 09:56:26.694090 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.694094 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.694098 | controller |     - '    logger.go:42: 09:41:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.694103 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.694107 | controller |     - '    logger.go:42: 09:41:16 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.694113 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.694117 | controller |       found'
2026-03-14 09:56:26.694121 | controller |     - '    logger.go:42: 09:41:17 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.694126 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.694130 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.694134 | controller |     - '        '
2026-03-14 09:56:26.694139 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.694143 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.694147 | controller |     - '        '
2026-03-14 09:56:26.694151 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.694163 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.694167 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.694172 | controller |     - '          exit 1'
2026-03-14 09:56:26.694176 | controller |     - '        fi'
2026-03-14 09:56:26.694180 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.694184 | controller |     - '        '
2026-03-14 09:56:26.694189 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.694193 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.694197 | controller |       | base64 -d)'
2026-03-14 09:56:26.694202 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.694206 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.694210 | controller |     - '          exit 1'
2026-03-14 09:56:26.694214 | controller |     - '        fi'
2026-03-14 09:56:26.694219 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.694223 | controller |     - '        '
2026-03-14 09:56:26.694227 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.694231 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.694236 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.694240 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.694244 | controller |       --timeout=300s'
2026-03-14 09:56:26.694248 | controller |     - '        '
2026-03-14 09:56:26.694253 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.694257 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.694261 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.694265 | controller |     - '        '
2026-03-14 09:56:26.694270 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.694274 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.694278 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.694282 | controller |     - '        '
2026-03-14 09:56:26.694287 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.694293 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.694297 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.694302 | controller |       available"'
2026-03-14 09:56:26.694306 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.694310 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.694316 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.694320 | controller |     - '          exit 1'
2026-03-14 09:56:26.694325 | controller |     - '        fi'
2026-03-14 09:56:26.694329 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.694333 | controller |     - '        '
2026-03-14 09:56:26.694339 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.694344 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.694349 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.694353 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.694358 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.694364 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.694368 | controller |       -- \'
2026-03-14 09:56:26.694372 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.694377 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.694381 | controller |     - '    logger.go:42: 09:41:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.694385 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.694389 | controller |     - '    logger.go:42: 09:41:18 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.694394 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.694398 | controller |       found'
2026-03-14 09:56:26.694402 | controller |     - '    logger.go:42: 09:41:19 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.694406 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.694411 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.694415 | controller |     - '        '
2026-03-14 09:56:26.694419 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.694423 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.694428 | controller |     - '        '
2026-03-14 09:56:26.694432 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.694436 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.694447 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.694451 | controller |     - '          exit 1'
2026-03-14 09:56:26.694456 | controller |     - '        fi'
2026-03-14 09:56:26.694460 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.694464 | controller |     - '        '
2026-03-14 09:56:26.694468 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.694473 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.694477 | controller |       | base64 -d)'
2026-03-14 09:56:26.694481 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.694485 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.694490 | controller |     - '          exit 1'
2026-03-14 09:56:26.694494 | controller |     - '        fi'
2026-03-14 09:56:26.694498 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.694502 | controller |     - '        '
2026-03-14 09:56:26.694507 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.694511 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.694515 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.694521 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.694526 | controller |       --timeout=300s'
2026-03-14 09:56:26.694530 | controller |     - '        '
2026-03-14 09:56:26.694534 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.694539 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.694543 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.694547 | controller |     - '        '
2026-03-14 09:56:26.694551 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.694556 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.694560 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.694564 | controller |     - '        '
2026-03-14 09:56:26.694568 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.694573 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.694577 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.694581 | controller |       available"'
2026-03-14 09:56:26.694585 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.694590 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.694608 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.694615 | controller |     - '          exit 1'
2026-03-14 09:56:26.694619 | controller |     - '        fi'
2026-03-14 09:56:26.694624 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.694628 | controller |     - '        '
2026-03-14 09:56:26.694632 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.694636 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.694643 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.694647 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.694651 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.694656 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.694662 | controller |       -- \'
2026-03-14 09:56:26.694666 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.694670 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.694674 | controller |     - '    logger.go:42: 09:41:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.694679 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.694683 | controller |     - '    logger.go:42: 09:41:19 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.694687 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.694691 | controller |       found'
2026-03-14 09:56:26.694696 | controller |     - '    logger.go:42: 09:41:20 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.694700 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.694704 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.694708 | controller |     - '        '
2026-03-14 09:56:26.694713 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.694719 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.694723 | controller |     - '        '
2026-03-14 09:56:26.694727 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.694731 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.694736 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.694746 | controller |     - '          exit 1'
2026-03-14 09:56:26.694751 | controller |     - '        fi'
2026-03-14 09:56:26.694755 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.694760 | controller |     - '        '
2026-03-14 09:56:26.694764 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.694768 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.694772 | controller |       | base64 -d)'
2026-03-14 09:56:26.694777 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.694781 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.694785 | controller |     - '          exit 1'
2026-03-14 09:56:26.694789 | controller |     - '        fi'
2026-03-14 09:56:26.694794 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.694798 | controller |     - '        '
2026-03-14 09:56:26.694802 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.694806 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.694810 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.694815 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.694819 | controller |       --timeout=300s'
2026-03-14 09:56:26.694823 | controller |     - '        '
2026-03-14 09:56:26.694827 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.694832 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.694836 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.694840 | controller |     - '        '
2026-03-14 09:56:26.694845 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.694849 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.694853 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.694857 | controller |     - '        '
2026-03-14 09:56:26.694862 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.694866 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.694870 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.694876 | controller |       available"'
2026-03-14 09:56:26.694880 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.694885 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.694889 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.694893 | controller |     - '          exit 1'
2026-03-14 09:56:26.694897 | controller |     - '        fi'
2026-03-14 09:56:26.694902 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.694906 | controller |     - '        '
2026-03-14 09:56:26.694910 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.694914 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.694920 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.694925 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.694929 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.694933 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.694937 | controller |       -- \'
2026-03-14 09:56:26.694942 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.694946 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.694950 | controller |     - '    logger.go:42: 09:41:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.694954 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.694959 | controller |     - '    logger.go:42: 09:41:20 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.694963 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.694967 | controller |       found'
2026-03-14 09:56:26.694971 | controller |     - '    logger.go:42: 09:41:21 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.694976 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.694992 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.694996 | controller |     - '        '
2026-03-14 09:56:26.695001 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.695005 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.695009 | controller |     - '        '
2026-03-14 09:56:26.695013 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.695018 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.695022 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.695026 | controller |     - '          exit 1'
2026-03-14 09:56:26.695031 | controller |     - '        fi'
2026-03-14 09:56:26.695042 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.695047 | controller |     - '        '
2026-03-14 09:56:26.695051 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.695055 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.695059 | controller |       | base64 -d)'
2026-03-14 09:56:26.695064 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.695068 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.695072 | controller |     - '          exit 1'
2026-03-14 09:56:26.695077 | controller |     - '        fi'
2026-03-14 09:56:26.695081 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.695085 | controller |     - '        '
2026-03-14 09:56:26.695089 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.695094 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.695098 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.695102 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.695106 | controller |       --timeout=300s'
2026-03-14 09:56:26.695111 | controller |     - '        '
2026-03-14 09:56:26.695117 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.695122 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.695126 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.695132 | controller |     - '        '
2026-03-14 09:56:26.695136 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.695140 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.695146 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.695151 | controller |     - '        '
2026-03-14 09:56:26.695155 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.695159 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.695163 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.695168 | controller |       available"'
2026-03-14 09:56:26.695172 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.695176 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.695182 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.695186 | controller |     - '          exit 1'
2026-03-14 09:56:26.695190 | controller |     - '        fi'
2026-03-14 09:56:26.695195 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.695199 | controller |     - '        '
2026-03-14 09:56:26.695203 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.695207 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.695212 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.695216 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.695220 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.695224 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.695229 | controller |       -- \'
2026-03-14 09:56:26.695233 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.695237 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.695242 | controller |     - '    logger.go:42: 09:41:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.695246 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.695250 | controller |     - '    logger.go:42: 09:41:21 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.695254 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.695259 | controller |       found'
2026-03-14 09:56:26.695264 | controller |     - '    logger.go:42: 09:41:22 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.695269 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.695273 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.695277 | controller |     - '        '
2026-03-14 09:56:26.695282 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.695286 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.695290 | controller |     - '        '
2026-03-14 09:56:26.695294 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.695299 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.695303 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.695307 | controller |     - '          exit 1'
2026-03-14 09:56:26.695311 | controller |     - '        fi'
2026-03-14 09:56:26.695319 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.695323 | controller |     - '        '
2026-03-14 09:56:26.695334 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.695339 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.695343 | controller |       | base64 -d)'
2026-03-14 09:56:26.695347 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.695351 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.695356 | controller |     - '          exit 1'
2026-03-14 09:56:26.695360 | controller |     - '        fi'
2026-03-14 09:56:26.695364 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.695368 | controller |     - '        '
2026-03-14 09:56:26.695373 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.695377 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.695381 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.695385 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.695390 | controller |       --timeout=300s'
2026-03-14 09:56:26.695394 | controller |     - '        '
2026-03-14 09:56:26.695398 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.695402 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.695407 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.695411 | controller |     - '        '
2026-03-14 09:56:26.695415 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.695419 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.695424 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.695428 | controller |     - '        '
2026-03-14 09:56:26.695432 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.695436 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.695441 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.695445 | controller |       available"'
2026-03-14 09:56:26.695449 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.695453 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.695458 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.695462 | controller |     - '          exit 1'
2026-03-14 09:56:26.695466 | controller |     - '        fi'
2026-03-14 09:56:26.695470 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.695477 | controller |     - '        '
2026-03-14 09:56:26.695481 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.695485 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.695490 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.695494 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.695498 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.695502 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.695516 | controller |       -- \'
2026-03-14 09:56:26.695521 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.695525 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.695529 | controller |     - '    logger.go:42: 09:41:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.695534 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.695538 | controller |     - '    logger.go:42: 09:41:22 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.695542 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.695546 | controller |       found'
2026-03-14 09:56:26.695551 | controller |     - '    logger.go:42: 09:41:23 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.695555 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.695559 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.695563 | controller |     - '        '
2026-03-14 09:56:26.695568 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.695572 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.695576 | controller |     - '        '
2026-03-14 09:56:26.695580 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.695585 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.695589 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.695603 | controller |     - '          exit 1'
2026-03-14 09:56:26.695610 | controller |     - '        fi'
2026-03-14 09:56:26.695614 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.695619 | controller |     - '        '
2026-03-14 09:56:26.695630 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.695634 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.695638 | controller |       | base64 -d)'
2026-03-14 09:56:26.695643 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.695647 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.695651 | controller |     - '          exit 1'
2026-03-14 09:56:26.695656 | controller |     - '        fi'
2026-03-14 09:56:26.695660 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.695664 | controller |     - '        '
2026-03-14 09:56:26.695668 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.695673 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.695677 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.695681 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.695685 | controller |       --timeout=300s'
2026-03-14 09:56:26.695690 | controller |     - '        '
2026-03-14 09:56:26.695694 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.695699 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.695705 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.695711 | controller |     - '        '
2026-03-14 09:56:26.695716 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.695722 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.695726 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.695731 | controller |     - '        '
2026-03-14 09:56:26.695737 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.695742 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.695746 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.695750 | controller |       available"'
2026-03-14 09:56:26.695755 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.695759 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.695763 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.695767 | controller |     - '          exit 1'
2026-03-14 09:56:26.695772 | controller |     - '        fi'
2026-03-14 09:56:26.695776 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.695780 | controller |     - '        '
2026-03-14 09:56:26.695784 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.695788 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.695793 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.695797 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.695801 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.695805 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.695810 | controller |       -- \'
2026-03-14 09:56:26.695814 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.695818 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.695822 | controller |     - '    logger.go:42: 09:41:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.695827 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.695831 | controller |     - '    logger.go:42: 09:41:23 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.695835 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.695840 | controller |       found'
2026-03-14 09:56:26.695844 | controller |     - '    logger.go:42: 09:41:24 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.695848 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.695852 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.695857 | controller |     - '        '
2026-03-14 09:56:26.695861 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.695865 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.695869 | controller |     - '        '
2026-03-14 09:56:26.695874 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.695878 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.695882 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.695886 | controller |     - '          exit 1'
2026-03-14 09:56:26.695891 | controller |     - '        fi'
2026-03-14 09:56:26.695895 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.695899 | controller |     - '        '
2026-03-14 09:56:26.695904 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.695914 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.695919 | controller |       | base64 -d)'
2026-03-14 09:56:26.695923 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.695927 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.695934 | controller |     - '          exit 1'
2026-03-14 09:56:26.695939 | controller |     - '        fi'
2026-03-14 09:56:26.695943 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.695947 | controller |     - '        '
2026-03-14 09:56:26.695951 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.695956 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.695960 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.695964 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.695968 | controller |       --timeout=300s'
2026-03-14 09:56:26.695972 | controller |     - '        '
2026-03-14 09:56:26.695977 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.695984 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.695988 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.695992 | controller |     - '        '
2026-03-14 09:56:26.695996 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.696001 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.696005 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.696009 | controller |     - '        '
2026-03-14 09:56:26.696013 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.696018 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.696022 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.696026 | controller |       available"'
2026-03-14 09:56:26.696030 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.696035 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.696039 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.696043 | controller |     - '          exit 1'
2026-03-14 09:56:26.696047 | controller |     - '        fi'
2026-03-14 09:56:26.696052 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.696056 | controller |     - '        '
2026-03-14 09:56:26.696060 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.696066 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.696070 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.696075 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.696079 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.696083 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.696087 | controller |       -- \'
2026-03-14 09:56:26.696092 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.696098 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.696102 | controller |     - '    logger.go:42: 09:41:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.696107 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.696111 | controller |     - '    logger.go:42: 09:41:25 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.696117 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.696121 | controller |       found'
2026-03-14 09:56:26.696125 | controller |     - '    logger.go:42: 09:41:26 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.696130 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.696134 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.696138 | controller |     - '        '
2026-03-14 09:56:26.696142 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.696147 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.696151 | controller |     - '        '
2026-03-14 09:56:26.696155 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.696159 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.696164 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.696168 | controller |     - '          exit 1'
2026-03-14 09:56:26.696172 | controller |     - '        fi'
2026-03-14 09:56:26.696176 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.696181 | controller |     - '        '
2026-03-14 09:56:26.696185 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.696195 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.696200 | controller |       | base64 -d)'
2026-03-14 09:56:26.696206 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.696210 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.696214 | controller |     - '          exit 1'
2026-03-14 09:56:26.696219 | controller |     - '        fi'
2026-03-14 09:56:26.696223 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.696227 | controller |     - '        '
2026-03-14 09:56:26.696232 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.696236 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.696240 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.696244 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.696248 | controller |       --timeout=300s'
2026-03-14 09:56:26.696253 | controller |     - '        '
2026-03-14 09:56:26.696257 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.696261 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.696266 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.696270 | controller |     - '        '
2026-03-14 09:56:26.696274 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.696278 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.696283 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.696287 | controller |     - '        '
2026-03-14 09:56:26.696291 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.696295 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.696299 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.696304 | controller |       available"'
2026-03-14 09:56:26.696308 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.696312 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.696318 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.696322 | controller |     - '          exit 1'
2026-03-14 09:56:26.696327 | controller |     - '        fi'
2026-03-14 09:56:26.696331 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.696335 | controller |     - '        '
2026-03-14 09:56:26.696339 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.696344 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.696348 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.696352 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.696356 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.696361 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.696365 | controller |       -- \'
2026-03-14 09:56:26.696369 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.696374 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.696378 | controller |     - '    logger.go:42: 09:41:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.696382 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.696386 | controller |     - '    logger.go:42: 09:41:26 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.696390 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.696395 | controller |       found'
2026-03-14 09:56:26.696399 | controller |     - '    logger.go:42: 09:41:27 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.696403 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.696409 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.696414 | controller |     - '        '
2026-03-14 09:56:26.696418 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.696422 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.696427 | controller |     - '        '
2026-03-14 09:56:26.696431 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.696436 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.696440 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.696444 | controller |     - '          exit 1'
2026-03-14 09:56:26.696449 | controller |     - '        fi'
2026-03-14 09:56:26.696453 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.696457 | controller |     - '        '
2026-03-14 09:56:26.696461 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.696467 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.696472 | controller |       | base64 -d)'
2026-03-14 09:56:26.696482 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.696487 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.696491 | controller |     - '          exit 1'
2026-03-14 09:56:26.696495 | controller |     - '        fi'
2026-03-14 09:56:26.696500 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.696504 | controller |     - '        '
2026-03-14 09:56:26.696508 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.696512 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.696516 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.696523 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.696527 | controller |       --timeout=300s'
2026-03-14 09:56:26.696531 | controller |     - '        '
2026-03-14 09:56:26.696535 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.696540 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.696544 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.696548 | controller |     - '        '
2026-03-14 09:56:26.696552 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.696557 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.696561 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.696565 | controller |     - '        '
2026-03-14 09:56:26.696570 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.696574 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.696578 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.696582 | controller |       available"'
2026-03-14 09:56:26.696587 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.696591 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.696606 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.696612 | controller |     - '          exit 1'
2026-03-14 09:56:26.696617 | controller |     - '        fi'
2026-03-14 09:56:26.696621 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.696625 | controller |     - '        '
2026-03-14 09:56:26.696630 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.696634 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.696638 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.696642 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.696653 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.696657 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.696662 | controller |       -- \'
2026-03-14 09:56:26.696666 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.696670 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.696674 | controller |     - '    logger.go:42: 09:41:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.696679 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.696683 | controller |     - '    logger.go:42: 09:41:27 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.696687 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.696692 | controller |       found'
2026-03-14 09:56:26.696696 | controller |     - '    logger.go:42: 09:41:28 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.696700 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.696704 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.696709 | controller |     - '        '
2026-03-14 09:56:26.696713 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.696717 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.696725 | controller |     - '        '
2026-03-14 09:56:26.696729 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.696733 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.696738 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.696742 | controller |     - '          exit 1'
2026-03-14 09:56:26.696746 | controller |     - '        fi'
2026-03-14 09:56:26.696750 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.696755 | controller |     - '        '
2026-03-14 09:56:26.696759 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.696763 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.696767 | controller |       | base64 -d)'
2026-03-14 09:56:26.696772 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.696783 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.696787 | controller |     - '          exit 1'
2026-03-14 09:56:26.696792 | controller |     - '        fi'
2026-03-14 09:56:26.696796 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.696800 | controller |     - '        '
2026-03-14 09:56:26.696804 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.696809 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.696813 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.696817 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.696821 | controller |       --timeout=300s'
2026-03-14 09:56:26.696826 | controller |     - '        '
2026-03-14 09:56:26.696830 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.696834 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.696838 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.696843 | controller |     - '        '
2026-03-14 09:56:26.696847 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.696851 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.696855 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.696860 | controller |     - '        '
2026-03-14 09:56:26.696864 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.696868 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.696872 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.696877 | controller |       available"'
2026-03-14 09:56:26.696881 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.696885 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.696889 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.696893 | controller |     - '          exit 1'
2026-03-14 09:56:26.696898 | controller |     - '        fi'
2026-03-14 09:56:26.696902 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.696906 | controller |     - '        '
2026-03-14 09:56:26.696911 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.696915 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.696919 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.696926 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.696930 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.696934 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.696939 | controller |       -- \'
2026-03-14 09:56:26.696943 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.696947 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.696951 | controller |     - '    logger.go:42: 09:41:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.696957 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.696963 | controller |     - '    logger.go:42: 09:41:28 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.696969 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.696974 | controller |       found'
2026-03-14 09:56:26.696979 | controller |     - '    logger.go:42: 09:41:29 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.696983 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.696987 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.696991 | controller |     - '        '
2026-03-14 09:56:26.696999 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.697005 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.697011 | controller |     - '        '
2026-03-14 09:56:26.697016 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.697021 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.697025 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.697029 | controller |     - '          exit 1'
2026-03-14 09:56:26.697034 | controller |     - '        fi'
2026-03-14 09:56:26.697038 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.697042 | controller |     - '        '
2026-03-14 09:56:26.697046 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.697051 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.697055 | controller |       | base64 -d)'
2026-03-14 09:56:26.697092 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.697107 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.697113 | controller |     - '          exit 1'
2026-03-14 09:56:26.697117 | controller |     - '        fi'
2026-03-14 09:56:26.697121 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.697126 | controller |     - '        '
2026-03-14 09:56:26.697130 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.697134 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.697139 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.697143 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.697147 | controller |       --timeout=300s'
2026-03-14 09:56:26.697152 | controller |     - '        '
2026-03-14 09:56:26.697156 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.697160 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.697165 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.697171 | controller |     - '        '
2026-03-14 09:56:26.697176 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.697180 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.697184 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.697191 | controller |     - '        '
2026-03-14 09:56:26.697195 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.697199 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.697204 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.697208 | controller |       available"'
2026-03-14 09:56:26.697212 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.697216 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.697221 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.697225 | controller |     - '          exit 1'
2026-03-14 09:56:26.697229 | controller |     - '        fi'
2026-03-14 09:56:26.697234 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.697238 | controller |     - '        '
2026-03-14 09:56:26.697242 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.697247 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.697251 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.697257 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.697261 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.697265 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.697270 | controller |       -- \'
2026-03-14 09:56:26.697274 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.697278 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.697282 | controller |     - '    logger.go:42: 09:41:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.697287 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.697293 | controller |     - '    logger.go:42: 09:41:29 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.697297 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.697301 | controller |       found'
2026-03-14 09:56:26.697306 | controller |     - '    logger.go:42: 09:41:30 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.697310 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.697314 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.697319 | controller |     - '        '
2026-03-14 09:56:26.697323 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.697327 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.697332 | controller |     - '        '
2026-03-14 09:56:26.697336 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.697340 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.697344 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.697349 | controller |     - '          exit 1'
2026-03-14 09:56:26.697353 | controller |     - '        fi'
2026-03-14 09:56:26.697357 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.697365 | controller |     - '        '
2026-03-14 09:56:26.697369 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.697373 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.697379 | controller |       | base64 -d)'
2026-03-14 09:56:26.697383 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.697388 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.697399 | controller |     - '          exit 1'
2026-03-14 09:56:26.697404 | controller |     - '        fi'
2026-03-14 09:56:26.697408 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.697412 | controller |     - '        '
2026-03-14 09:56:26.697417 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.697421 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.697425 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.697429 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.697434 | controller |       --timeout=300s'
2026-03-14 09:56:26.697438 | controller |     - '        '
2026-03-14 09:56:26.697442 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.697446 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.697451 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.697455 | controller |     - '        '
2026-03-14 09:56:26.697459 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.697464 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.697468 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.697472 | controller |     - '        '
2026-03-14 09:56:26.697476 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.697481 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.697485 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.697489 | controller |       available"'
2026-03-14 09:56:26.697494 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.697498 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.697502 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.697506 | controller |     - '          exit 1'
2026-03-14 09:56:26.697511 | controller |     - '        fi'
2026-03-14 09:56:26.697515 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.697519 | controller |     - '        '
2026-03-14 09:56:26.697523 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.697528 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.697532 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.697536 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.697540 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.697545 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.697549 | controller |       -- \'
2026-03-14 09:56:26.697553 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.697559 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.697564 | controller |     - '    logger.go:42: 09:41:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.697568 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.697572 | controller |     - '    logger.go:42: 09:41:30 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.697577 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.697581 | controller |       found'
2026-03-14 09:56:26.697585 | controller |     - '    logger.go:42: 09:41:31 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.697589 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.697605 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.697612 | controller |     - '        '
2026-03-14 09:56:26.697624 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.697629 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.697633 | controller |     - '        '
2026-03-14 09:56:26.697637 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.697642 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.697646 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.697650 | controller |     - '          exit 1'
2026-03-14 09:56:26.697655 | controller |     - '        fi'
2026-03-14 09:56:26.697659 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.697663 | controller |     - '        '
2026-03-14 09:56:26.697667 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.697672 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.697676 | controller |       | base64 -d)'
2026-03-14 09:56:26.697680 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.697684 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.697689 | controller |     - '          exit 1'
2026-03-14 09:56:26.697693 | controller |     - '        fi'
2026-03-14 09:56:26.697704 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.697709 | controller |     - '        '
2026-03-14 09:56:26.697713 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.697718 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.697722 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.697726 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.697730 | controller |       --timeout=300s'
2026-03-14 09:56:26.697734 | controller |     - '        '
2026-03-14 09:56:26.697739 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.697743 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.697747 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.697752 | controller |     - '        '
2026-03-14 09:56:26.697756 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.697760 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.697764 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.697769 | controller |     - '        '
2026-03-14 09:56:26.697773 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.697779 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.697784 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.697788 | controller |       available"'
2026-03-14 09:56:26.697792 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.697797 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.697803 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.697807 | controller |     - '          exit 1'
2026-03-14 09:56:26.697811 | controller |     - '        fi'
2026-03-14 09:56:26.697816 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.697820 | controller |     - '        '
2026-03-14 09:56:26.697824 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.697828 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.697833 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.697837 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.697841 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.697846 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.697850 | controller |       -- \'
2026-03-14 09:56:26.697854 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.697858 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.697862 | controller |     - '    logger.go:42: 09:41:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.697867 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.697871 | controller |     - '    logger.go:42: 09:41:32 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.697877 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.697881 | controller |       found'
2026-03-14 09:56:26.697885 | controller |     - '    logger.go:42: 09:41:33 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.697890 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.697894 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.697898 | controller |     - '        '
2026-03-14 09:56:26.697904 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.697910 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.697916 | controller |     - '        '
2026-03-14 09:56:26.697922 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.697926 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.697930 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.697935 | controller |     - '          exit 1'
2026-03-14 09:56:26.697939 | controller |     - '        fi'
2026-03-14 09:56:26.697943 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.697947 | controller |     - '        '
2026-03-14 09:56:26.697952 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.697956 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.697960 | controller |       | base64 -d)'
2026-03-14 09:56:26.697964 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.697969 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.697973 | controller |     - '          exit 1'
2026-03-14 09:56:26.697979 | controller |     - '        fi'
2026-03-14 09:56:26.697984 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.697995 | controller |     - '        '
2026-03-14 09:56:26.697999 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.698004 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.698008 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.698012 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.698017 | controller |       --timeout=300s'
2026-03-14 09:56:26.698021 | controller |     - '        '
2026-03-14 09:56:26.698025 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.698029 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.698034 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.698038 | controller |     - '        '
2026-03-14 09:56:26.698042 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.698046 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.698051 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.698055 | controller |     - '        '
2026-03-14 09:56:26.698059 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.698064 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.698068 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.698072 | controller |       available"'
2026-03-14 09:56:26.698076 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.698081 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.698085 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.698089 | controller |     - '          exit 1'
2026-03-14 09:56:26.698093 | controller |     - '        fi'
2026-03-14 09:56:26.698097 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.698102 | controller |     - '        '
2026-03-14 09:56:26.698106 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.698110 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.698114 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.698119 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.698123 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.698127 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.698131 | controller |       -- \'
2026-03-14 09:56:26.698136 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.698140 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.698144 | controller |     - '    logger.go:42: 09:41:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.698149 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.698153 | controller |     - '    logger.go:42: 09:41:33 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.698157 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.698165 | controller |       found'
2026-03-14 09:56:26.698169 | controller |     - '    logger.go:42: 09:41:34 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.698173 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.698177 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.698182 | controller |     - '        '
2026-03-14 09:56:26.698186 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.698190 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.698195 | controller |     - '        '
2026-03-14 09:56:26.698199 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.698203 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.698207 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.698212 | controller |     - '          exit 1'
2026-03-14 09:56:26.698216 | controller |     - '        fi'
2026-03-14 09:56:26.698220 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.698225 | controller |     - '        '
2026-03-14 09:56:26.698229 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.698233 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.698237 | controller |       | base64 -d)'
2026-03-14 09:56:26.698242 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.698246 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.698250 | controller |     - '          exit 1'
2026-03-14 09:56:26.698254 | controller |     - '        fi'
2026-03-14 09:56:26.698259 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.698263 | controller |     - '        '
2026-03-14 09:56:26.698273 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.698278 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.698282 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.698287 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.698291 | controller |       --timeout=300s'
2026-03-14 09:56:26.698295 | controller |     - '        '
2026-03-14 09:56:26.698300 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.698304 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.698310 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.698315 | controller |     - '        '
2026-03-14 09:56:26.698319 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.698323 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.698328 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.698334 | controller |     - '        '
2026-03-14 09:56:26.698338 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.698342 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.698347 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.698351 | controller |       available"'
2026-03-14 09:56:26.698355 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.698359 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.698364 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.698369 | controller |     - '          exit 1'
2026-03-14 09:56:26.698374 | controller |     - '        fi'
2026-03-14 09:56:26.698378 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.698382 | controller |     - '        '
2026-03-14 09:56:26.698386 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.698391 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.698395 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.698399 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.698404 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.698408 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.698412 | controller |       -- \'
2026-03-14 09:56:26.698416 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.698421 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.698425 | controller |     - '    logger.go:42: 09:41:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.698429 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.698434 | controller |     - '    logger.go:42: 09:41:34 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.698438 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.698444 | controller |       found'
2026-03-14 09:56:26.698448 | controller |     - '    logger.go:42: 09:41:35 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.698452 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.698457 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.698461 | controller |     - '        '
2026-03-14 09:56:26.698465 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.698469 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.698474 | controller |     - '        '
2026-03-14 09:56:26.698478 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.698482 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.698486 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.698491 | controller |     - '          exit 1'
2026-03-14 09:56:26.698495 | controller |     - '        fi'
2026-03-14 09:56:26.698499 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.698503 | controller |     - '        '
2026-03-14 09:56:26.698508 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.698512 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.698516 | controller |       | base64 -d)'
2026-03-14 09:56:26.698520 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.698525 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.698529 | controller |     - '          exit 1'
2026-03-14 09:56:26.698533 | controller |     - '        fi'
2026-03-14 09:56:26.698537 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.698542 | controller |     - '        '
2026-03-14 09:56:26.698546 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.698556 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.698561 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.698565 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.698571 | controller |       --timeout=300s'
2026-03-14 09:56:26.698575 | controller |     - '        '
2026-03-14 09:56:26.698579 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.698584 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.698588 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.698592 | controller |     - '        '
2026-03-14 09:56:26.698611 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.698615 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.698619 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.698624 | controller |     - '        '
2026-03-14 09:56:26.698628 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.698632 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.698636 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.698641 | controller |       available"'
2026-03-14 09:56:26.698645 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.698649 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.698653 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.698658 | controller |     - '          exit 1'
2026-03-14 09:56:26.698662 | controller |     - '        fi'
2026-03-14 09:56:26.698666 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.698671 | controller |     - '        '
2026-03-14 09:56:26.698675 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.698679 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.698683 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.698688 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.698692 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.698696 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.698703 | controller |       -- \'
2026-03-14 09:56:26.698707 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.698711 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.698715 | controller |     - '    logger.go:42: 09:41:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.698720 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.698724 | controller |     - '    logger.go:42: 09:41:35 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.698728 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.698733 | controller |       found'
2026-03-14 09:56:26.698737 | controller |     - '    logger.go:42: 09:41:36 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.698741 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.698745 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.698750 | controller |     - '        '
2026-03-14 09:56:26.698754 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.698758 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.698764 | controller |     - '        '
2026-03-14 09:56:26.698768 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.698773 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.698777 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.698781 | controller |     - '          exit 1'
2026-03-14 09:56:26.698785 | controller |     - '        fi'
2026-03-14 09:56:26.698790 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.698794 | controller |     - '        '
2026-03-14 09:56:26.698798 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.698803 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.698807 | controller |       | base64 -d)'
2026-03-14 09:56:26.698811 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.698815 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.698820 | controller |     - '          exit 1'
2026-03-14 09:56:26.698824 | controller |     - '        fi'
2026-03-14 09:56:26.698828 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.698832 | controller |     - '        '
2026-03-14 09:56:26.698837 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.698841 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.698852 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.698856 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.698861 | controller |       --timeout=300s'
2026-03-14 09:56:26.698865 | controller |     - '        '
2026-03-14 09:56:26.698869 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.698874 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.698878 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.698882 | controller |     - '        '
2026-03-14 09:56:26.698886 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.698891 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.698895 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.698899 | controller |     - '        '
2026-03-14 09:56:26.698904 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.698908 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.698912 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.698916 | controller |       available"'
2026-03-14 09:56:26.698921 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.698925 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.698929 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.698933 | controller |     - '          exit 1'
2026-03-14 09:56:26.698938 | controller |     - '        fi'
2026-03-14 09:56:26.698942 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.698946 | controller |     - '        '
2026-03-14 09:56:26.698950 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.698955 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.698959 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.698965 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.698969 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.698974 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.698978 | controller |       -- \'
2026-03-14 09:56:26.698994 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.699000 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.699005 | controller |     - '    logger.go:42: 09:41:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.699009 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.699013 | controller |     - '    logger.go:42: 09:41:36 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.699018 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.699022 | controller |       found'
2026-03-14 09:56:26.699026 | controller |     - '    logger.go:42: 09:41:37 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.699033 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.699037 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.699041 | controller |     - '        '
2026-03-14 09:56:26.699045 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.699050 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.699054 | controller |     - '        '
2026-03-14 09:56:26.699058 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.699062 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.699066 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.699071 | controller |     - '          exit 1'
2026-03-14 09:56:26.699075 | controller |     - '        fi'
2026-03-14 09:56:26.699079 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.699083 | controller |     - '        '
2026-03-14 09:56:26.699088 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.699093 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.699098 | controller |       | base64 -d)'
2026-03-14 09:56:26.699102 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.699106 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.699110 | controller |     - '          exit 1'
2026-03-14 09:56:26.699115 | controller |     - '        fi'
2026-03-14 09:56:26.699119 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.699123 | controller |     - '        '
2026-03-14 09:56:26.699127 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.699131 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.699142 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.699147 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.699151 | controller |       --timeout=300s'
2026-03-14 09:56:26.699155 | controller |     - '        '
2026-03-14 09:56:26.699160 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.699164 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.699168 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.699172 | controller |     - '        '
2026-03-14 09:56:26.699179 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.699183 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.699187 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.699192 | controller |     - '        '
2026-03-14 09:56:26.699196 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.699200 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.699205 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.699209 | controller |       available"'
2026-03-14 09:56:26.699213 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.699217 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.699222 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.699226 | controller |     - '          exit 1'
2026-03-14 09:56:26.699230 | controller |     - '        fi'
2026-03-14 09:56:26.699234 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.699238 | controller |     - '        '
2026-03-14 09:56:26.699243 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.699247 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.699251 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.699257 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.699262 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.699266 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.699270 | controller |       -- \'
2026-03-14 09:56:26.699274 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.699279 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.699283 | controller |     - '    logger.go:42: 09:41:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.699287 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.699291 | controller |     - '    logger.go:42: 09:41:37 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.699296 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.699300 | controller |       found'
2026-03-14 09:56:26.699304 | controller |     - '    logger.go:42: 09:41:38 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.699308 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.699312 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.699317 | controller |     - '        '
2026-03-14 09:56:26.699321 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.699325 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.699329 | controller |     - '        '
2026-03-14 09:56:26.699334 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.699338 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.699342 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.699346 | controller |     - '          exit 1'
2026-03-14 09:56:26.699351 | controller |     - '        fi'
2026-03-14 09:56:26.699355 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.699359 | controller |     - '        '
2026-03-14 09:56:26.699365 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.699370 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.699374 | controller |       | base64 -d)'
2026-03-14 09:56:26.699378 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.699382 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.699387 | controller |     - '          exit 1'
2026-03-14 09:56:26.699392 | controller |     - '        fi'
2026-03-14 09:56:26.699400 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.699404 | controller |     - '        '
2026-03-14 09:56:26.699408 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.699412 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.699417 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.699427 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.699432 | controller |       --timeout=300s'
2026-03-14 09:56:26.699436 | controller |     - '        '
2026-03-14 09:56:26.699440 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.699445 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.699449 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.699453 | controller |     - '        '
2026-03-14 09:56:26.699457 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.699462 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.699466 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.699470 | controller |     - '        '
2026-03-14 09:56:26.699474 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.699479 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.699483 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.699487 | controller |       available"'
2026-03-14 09:56:26.699491 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.699496 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.699500 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.699504 | controller |     - '          exit 1'
2026-03-14 09:56:26.699508 | controller |     - '        fi'
2026-03-14 09:56:26.699513 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.699517 | controller |     - '        '
2026-03-14 09:56:26.699521 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.699525 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.699530 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.699534 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.699538 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.699542 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.699547 | controller |       -- \'
2026-03-14 09:56:26.699551 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.699557 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.699561 | controller |     - '    logger.go:42: 09:41:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.699566 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.699570 | controller |     - '    logger.go:42: 09:41:39 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.699574 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.699578 | controller |       found'
2026-03-14 09:56:26.699583 | controller |     - '    logger.go:42: 09:41:40 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.699587 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.699591 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.699606 | controller |     - '        '
2026-03-14 09:56:26.699612 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.699628 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.699632 | controller |     - '        '
2026-03-14 09:56:26.699636 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.699641 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.699645 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.699649 | controller |     - '          exit 1'
2026-03-14 09:56:26.699653 | controller |     - '        fi'
2026-03-14 09:56:26.699658 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.699662 | controller |     - '        '
2026-03-14 09:56:26.699666 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.699670 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.699675 | controller |       | base64 -d)'
2026-03-14 09:56:26.699679 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.699683 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.699687 | controller |     - '          exit 1'
2026-03-14 09:56:26.699692 | controller |     - '        fi'
2026-03-14 09:56:26.699696 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.699700 | controller |     - '        '
2026-03-14 09:56:26.699704 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.699709 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.699713 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.699725 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.699729 | controller |       --timeout=300s'
2026-03-14 09:56:26.699734 | controller |     - '        '
2026-03-14 09:56:26.699738 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.699742 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.699747 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.699751 | controller |     - '        '
2026-03-14 09:56:26.699755 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.699759 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.699764 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.699768 | controller |     - '        '
2026-03-14 09:56:26.699772 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.699779 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.699783 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.699787 | controller |       available"'
2026-03-14 09:56:26.699792 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.699796 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.699800 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.699804 | controller |     - '          exit 1'
2026-03-14 09:56:26.699809 | controller |     - '        fi'
2026-03-14 09:56:26.699813 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.699817 | controller |     - '        '
2026-03-14 09:56:26.699821 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.699826 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.699830 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.699834 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.699838 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.699843 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.699847 | controller |       -- \'
2026-03-14 09:56:26.699851 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.699855 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.699860 | controller |     - '    logger.go:42: 09:41:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.699864 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.699868 | controller |     - '    logger.go:42: 09:41:40 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.699872 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.699876 | controller |       found'
2026-03-14 09:56:26.699881 | controller |     - '    logger.go:42: 09:41:41 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.699885 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.699889 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.699894 | controller |     - '        '
2026-03-14 09:56:26.699898 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.699902 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.699906 | controller |     - '        '
2026-03-14 09:56:26.699911 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.699915 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.699921 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.699925 | controller |     - '          exit 1'
2026-03-14 09:56:26.699930 | controller |     - '        fi'
2026-03-14 09:56:26.699934 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.699938 | controller |     - '        '
2026-03-14 09:56:26.699943 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.699947 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.699951 | controller |       | base64 -d)'
2026-03-14 09:56:26.699956 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.699960 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.699964 | controller |     - '          exit 1'
2026-03-14 09:56:26.699968 | controller |     - '        fi'
2026-03-14 09:56:26.699973 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.699979 | controller |     - '        '
2026-03-14 09:56:26.699983 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.699987 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.699992 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.699996 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.700000 | controller |       --timeout=300s'
2026-03-14 09:56:26.700004 | controller |     - '        '
2026-03-14 09:56:26.700015 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.700022 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.700026 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.700030 | controller |     - '        '
2026-03-14 09:56:26.700035 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.700039 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.700043 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.700048 | controller |     - '        '
2026-03-14 09:56:26.700052 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.700056 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.700061 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.700065 | controller |       available"'
2026-03-14 09:56:26.700069 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.700073 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.700078 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.700082 | controller |     - '          exit 1'
2026-03-14 09:56:26.700086 | controller |     - '        fi'
2026-03-14 09:56:26.700091 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.700095 | controller |     - '        '
2026-03-14 09:56:26.700100 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.700106 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.700112 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.700118 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.700123 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.700128 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.700132 | controller |       -- \'
2026-03-14 09:56:26.700136 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.700141 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.700145 | controller |     - '    logger.go:42: 09:41:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.700149 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.700154 | controller |     - '    logger.go:42: 09:41:41 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.700158 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.700165 | controller |       found'
2026-03-14 09:56:26.700170 | controller |     - '    logger.go:42: 09:41:42 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.700174 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.700178 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.700183 | controller |     - '        '
2026-03-14 09:56:26.700187 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.700191 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.700198 | controller |     - '        '
2026-03-14 09:56:26.700202 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.700206 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.700210 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.700215 | controller |     - '          exit 1'
2026-03-14 09:56:26.700219 | controller |     - '        fi'
2026-03-14 09:56:26.700223 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.700227 | controller |     - '        '
2026-03-14 09:56:26.700232 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.700236 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.700240 | controller |       | base64 -d)'
2026-03-14 09:56:26.700244 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.700249 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.700253 | controller |     - '          exit 1'
2026-03-14 09:56:26.700257 | controller |     - '        fi'
2026-03-14 09:56:26.700261 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.700266 | controller |     - '        '
2026-03-14 09:56:26.700270 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.700274 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.700278 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.700283 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.700287 | controller |       --timeout=300s'
2026-03-14 09:56:26.700291 | controller |     - '        '
2026-03-14 09:56:26.700302 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.700307 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.700311 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.700318 | controller |     - '        '
2026-03-14 09:56:26.700322 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.700326 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.700331 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.700336 | controller |     - '        '
2026-03-14 09:56:26.700341 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.700346 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.700350 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.700355 | controller |       available"'
2026-03-14 09:56:26.700361 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.700366 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.700372 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.700376 | controller |     - '          exit 1'
2026-03-14 09:56:26.700382 | controller |     - '        fi'
2026-03-14 09:56:26.700386 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.700391 | controller |     - '        '
2026-03-14 09:56:26.700395 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.700399 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.700403 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.700407 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.700412 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.700416 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.700420 | controller |       -- \'
2026-03-14 09:56:26.700424 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.700429 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.700433 | controller |     - '    logger.go:42: 09:41:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.700437 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.700442 | controller |     - '    logger.go:42: 09:41:42 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.700446 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.700450 | controller |       found'
2026-03-14 09:56:26.700454 | controller |     - '    logger.go:42: 09:41:43 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.700459 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.700463 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.700467 | controller |     - '        '
2026-03-14 09:56:26.700472 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.700476 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.700480 | controller |     - '        '
2026-03-14 09:56:26.700484 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.700489 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.700493 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.700497 | controller |     - '          exit 1'
2026-03-14 09:56:26.700503 | controller |     - '        fi'
2026-03-14 09:56:26.700507 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.700512 | controller |     - '        '
2026-03-14 09:56:26.700516 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.700520 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.700524 | controller |       | base64 -d)'
2026-03-14 09:56:26.700529 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.700533 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.700537 | controller |     - '          exit 1'
2026-03-14 09:56:26.700541 | controller |     - '        fi'
2026-03-14 09:56:26.700546 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.700550 | controller |     - '        '
2026-03-14 09:56:26.700554 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.700558 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.700562 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.700567 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.700571 | controller |       --timeout=300s'
2026-03-14 09:56:26.700577 | controller |     - '        '
2026-03-14 09:56:26.700581 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.700591 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.700609 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.700614 | controller |     - '        '
2026-03-14 09:56:26.700618 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.700622 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.700627 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.700631 | controller |     - '        '
2026-03-14 09:56:26.700635 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.700639 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.700644 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.700648 | controller |       available"'
2026-03-14 09:56:26.700652 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.700656 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.700661 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.700665 | controller |     - '          exit 1'
2026-03-14 09:56:26.700669 | controller |     - '        fi'
2026-03-14 09:56:26.700673 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.700678 | controller |     - '        '
2026-03-14 09:56:26.700682 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.700686 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.700690 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.700695 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.700699 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.700703 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.700707 | controller |       -- \'
2026-03-14 09:56:26.700712 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.700716 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.700720 | controller |     - '    logger.go:42: 09:41:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.700724 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.700729 | controller |     - '    logger.go:42: 09:41:43 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.700733 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.700737 | controller |       found'
2026-03-14 09:56:26.700742 | controller |     - '    logger.go:42: 09:41:44 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.700746 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.700750 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.700754 | controller |     - '        '
2026-03-14 09:56:26.700759 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.700763 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.700767 | controller |     - '        '
2026-03-14 09:56:26.700773 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.700781 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.700785 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.700789 | controller |     - '          exit 1'
2026-03-14 09:56:26.700794 | controller |     - '        fi'
2026-03-14 09:56:26.700798 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.700802 | controller |     - '        '
2026-03-14 09:56:26.700806 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.700811 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.700815 | controller |       | base64 -d)'
2026-03-14 09:56:26.700819 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.700823 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.700828 | controller |     - '          exit 1'
2026-03-14 09:56:26.700832 | controller |     - '        fi'
2026-03-14 09:56:26.700836 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.700840 | controller |     - '        '
2026-03-14 09:56:26.700845 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.700849 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.700853 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.700857 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.700862 | controller |       --timeout=300s'
2026-03-14 09:56:26.700866 | controller |     - '        '
2026-03-14 09:56:26.700870 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.700881 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.700885 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.700890 | controller |     - '        '
2026-03-14 09:56:26.700894 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.700898 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.700902 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.700907 | controller |     - '        '
2026-03-14 09:56:26.700911 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.700915 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.700920 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.700924 | controller |       available"'
2026-03-14 09:56:26.700928 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.700932 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.700937 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.700941 | controller |     - '          exit 1'
2026-03-14 09:56:26.700945 | controller |     - '        fi'
2026-03-14 09:56:26.700949 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.700954 | controller |     - '        '
2026-03-14 09:56:26.700958 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.700962 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.700966 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.700972 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.700977 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.700981 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.700985 | controller |       -- \'
2026-03-14 09:56:26.700990 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.700994 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.700998 | controller |     - '    logger.go:42: 09:41:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.701002 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.701007 | controller |     - '    logger.go:42: 09:41:44 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.701011 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.701015 | controller |       found'
2026-03-14 09:56:26.701020 | controller |     - '    logger.go:42: 09:41:45 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.701024 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.701028 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.701032 | controller |     - '        '
2026-03-14 09:56:26.701037 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.701041 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.701045 | controller |     - '        '
2026-03-14 09:56:26.701049 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.701054 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.701058 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.701062 | controller |     - '          exit 1'
2026-03-14 09:56:26.701066 | controller |     - '        fi'
2026-03-14 09:56:26.701071 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.701075 | controller |     - '        '
2026-03-14 09:56:26.701079 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.701083 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.701088 | controller |       | base64 -d)'
2026-03-14 09:56:26.701092 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.701096 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.701100 | controller |     - '          exit 1'
2026-03-14 09:56:26.701105 | controller |     - '        fi'
2026-03-14 09:56:26.701109 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.701113 | controller |     - '        '
2026-03-14 09:56:26.701117 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.701122 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.701126 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.701130 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.701134 | controller |       --timeout=300s'
2026-03-14 09:56:26.701139 | controller |     - '        '
2026-03-14 09:56:26.701143 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.701153 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.701158 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.701162 | controller |     - '        '
2026-03-14 09:56:26.701166 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.701172 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.701176 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.701181 | controller |     - '        '
2026-03-14 09:56:26.701185 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.701189 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.701194 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.701198 | controller |       available"'
2026-03-14 09:56:26.701202 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.701206 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.701210 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.701215 | controller |     - '          exit 1'
2026-03-14 09:56:26.701219 | controller |     - '        fi'
2026-03-14 09:56:26.701225 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.701231 | controller |     - '        '
2026-03-14 09:56:26.701237 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.701243 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.701249 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.701255 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.701260 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.701267 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.701272 | controller |       -- \'
2026-03-14 09:56:26.701276 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.701280 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.701285 | controller |     - '    logger.go:42: 09:41:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.701289 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.701293 | controller |     - '    logger.go:42: 09:41:45 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.701297 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.701302 | controller |       found'
2026-03-14 09:56:26.701306 | controller |     - '    logger.go:42: 09:41:46 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.701310 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.701315 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.701319 | controller |     - '        '
2026-03-14 09:56:26.701323 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.701327 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.701332 | controller |     - '        '
2026-03-14 09:56:26.701336 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.701340 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.701345 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.701349 | controller |     - '          exit 1'
2026-03-14 09:56:26.701353 | controller |     - '        fi'
2026-03-14 09:56:26.701358 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.701364 | controller |     - '        '
2026-03-14 09:56:26.701368 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.701382 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.701388 | controller |       | base64 -d)'
2026-03-14 09:56:26.701392 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.701397 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.701401 | controller |     - '          exit 1'
2026-03-14 09:56:26.701405 | controller |     - '        fi'
2026-03-14 09:56:26.701410 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.701414 | controller |     - '        '
2026-03-14 09:56:26.701418 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.701422 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.701428 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.701432 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.701437 | controller |       --timeout=300s'
2026-03-14 09:56:26.701441 | controller |     - '        '
2026-03-14 09:56:26.701445 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.701449 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.701462 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.701466 | controller |     - '        '
2026-03-14 09:56:26.701471 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.701475 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.701479 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.701483 | controller |     - '        '
2026-03-14 09:56:26.701488 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.701492 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.701496 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.701501 | controller |       available"'
2026-03-14 09:56:26.701505 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.701509 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.701514 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.701518 | controller |     - '          exit 1'
2026-03-14 09:56:26.701525 | controller |     - '        fi'
2026-03-14 09:56:26.701529 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.701533 | controller |     - '        '
2026-03-14 09:56:26.701538 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.701542 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.701546 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.701550 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.701555 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.701559 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.701563 | controller |       -- \'
2026-03-14 09:56:26.701568 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.701572 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.701578 | controller |     - '    logger.go:42: 09:41:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.701582 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.701587 | controller |     - '    logger.go:42: 09:41:47 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.701591 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.701612 | controller |       found'
2026-03-14 09:56:26.701617 | controller |     - '    logger.go:42: 09:41:48 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.701621 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.701626 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.701630 | controller |     - '        '
2026-03-14 09:56:26.701634 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.701639 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.701643 | controller |     - '        '
2026-03-14 09:56:26.701647 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.701651 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.701656 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.701660 | controller |     - '          exit 1'
2026-03-14 09:56:26.701664 | controller |     - '        fi'
2026-03-14 09:56:26.701669 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.701673 | controller |     - '        '
2026-03-14 09:56:26.701677 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.701683 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.701688 | controller |       | base64 -d)'
2026-03-14 09:56:26.701692 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.701696 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.701701 | controller |     - '          exit 1'
2026-03-14 09:56:26.701705 | controller |     - '        fi'
2026-03-14 09:56:26.701709 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.701713 | controller |     - '        '
2026-03-14 09:56:26.701718 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.701722 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.701726 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.701731 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.701735 | controller |       --timeout=300s'
2026-03-14 09:56:26.701739 | controller |     - '        '
2026-03-14 09:56:26.701743 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.701748 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.701758 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.701763 | controller |     - '        '
2026-03-14 09:56:26.701767 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.701772 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.701776 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.701782 | controller |     - '        '
2026-03-14 09:56:26.701786 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.701790 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.701797 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.701802 | controller |       available"'
2026-03-14 09:56:26.701806 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.701810 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.701815 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.701819 | controller |     - '          exit 1'
2026-03-14 09:56:26.701823 | controller |     - '        fi'
2026-03-14 09:56:26.701827 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.701832 | controller |     - '        '
2026-03-14 09:56:26.701836 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.701840 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.701845 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.701849 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.701853 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.701857 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.701862 | controller |       -- \'
2026-03-14 09:56:26.701866 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.701870 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.701875 | controller |     - '    logger.go:42: 09:41:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.701879 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.701883 | controller |     - '    logger.go:42: 09:41:48 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.701887 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.701892 | controller |       found'
2026-03-14 09:56:26.701896 | controller |     - '    logger.go:42: 09:41:49 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.701900 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.701905 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.701909 | controller |     - '        '
2026-03-14 09:56:26.701913 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.701917 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.701922 | controller |     - '        '
2026-03-14 09:56:26.701926 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.701930 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.701935 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.701939 | controller |     - '          exit 1'
2026-03-14 09:56:26.701943 | controller |     - '        fi'
2026-03-14 09:56:26.701947 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.701952 | controller |     - '        '
2026-03-14 09:56:26.701956 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.701960 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.701965 | controller |       | base64 -d)'
2026-03-14 09:56:26.701969 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.701973 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.701977 | controller |     - '          exit 1'
2026-03-14 09:56:26.701982 | controller |     - '        fi'
2026-03-14 09:56:26.701986 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.701992 | controller |     - '        '
2026-03-14 09:56:26.701996 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.702001 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.702005 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.702011 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.702015 | controller |       --timeout=300s'
2026-03-14 09:56:26.702019 | controller |     - '        '
2026-03-14 09:56:26.702024 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.702028 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.702032 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.702037 | controller |     - '        '
2026-03-14 09:56:26.702047 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.702051 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.702056 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.702060 | controller |     - '        '
2026-03-14 09:56:26.702064 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.702069 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.702073 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.702077 | controller |       available"'
2026-03-14 09:56:26.702082 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.702086 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.702090 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.702094 | controller |     - '          exit 1'
2026-03-14 09:56:26.702099 | controller |     - '        fi'
2026-03-14 09:56:26.702103 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.702107 | controller |     - '        '
2026-03-14 09:56:26.702112 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.702116 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.702120 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.702125 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.702129 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.702133 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.702137 | controller |       -- \'
2026-03-14 09:56:26.702142 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.702146 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.702150 | controller |     - '    logger.go:42: 09:41:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.702155 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.702159 | controller |     - '    logger.go:42: 09:41:49 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.702163 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.702167 | controller |       found'
2026-03-14 09:56:26.702172 | controller |     - '    logger.go:42: 09:41:50 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.702179 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.702183 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.702188 | controller |     - '        '
2026-03-14 09:56:26.702192 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.702196 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.702200 | controller |     - '        '
2026-03-14 09:56:26.702205 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.702209 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.702213 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.702218 | controller |     - '          exit 1'
2026-03-14 09:56:26.702222 | controller |     - '        fi'
2026-03-14 09:56:26.702226 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.702230 | controller |     - '        '
2026-03-14 09:56:26.702235 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.702239 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.702243 | controller |       | base64 -d)'
2026-03-14 09:56:26.702247 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.702252 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.702256 | controller |     - '          exit 1'
2026-03-14 09:56:26.702260 | controller |     - '        fi'
2026-03-14 09:56:26.702265 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.702269 | controller |     - '        '
2026-03-14 09:56:26.702273 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.702277 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.702282 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.702286 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.702290 | controller |       --timeout=300s'
2026-03-14 09:56:26.702294 | controller |     - '        '
2026-03-14 09:56:26.702299 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.702304 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.702310 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.702316 | controller |     - '        '
2026-03-14 09:56:26.702329 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.702333 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.702340 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.702344 | controller |     - '        '
2026-03-14 09:56:26.702349 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.702353 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.702357 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.702362 | controller |       available"'
2026-03-14 09:56:26.702366 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.702370 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.702374 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.702379 | controller |     - '          exit 1'
2026-03-14 09:56:26.702383 | controller |     - '        fi'
2026-03-14 09:56:26.702389 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.702393 | controller |     - '        '
2026-03-14 09:56:26.702398 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.702402 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.702406 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.702410 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.702415 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.702419 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.702423 | controller |       -- \'
2026-03-14 09:56:26.702428 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.702432 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.702436 | controller |     - '    logger.go:42: 09:41:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.702441 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.702445 | controller |     - '    logger.go:42: 09:41:50 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.702449 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.702453 | controller |       found'
2026-03-14 09:56:26.702458 | controller |     - '    logger.go:42: 09:41:51 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.702462 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.702466 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.702471 | controller |     - '        '
2026-03-14 09:56:26.702475 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.702479 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.702484 | controller |     - '        '
2026-03-14 09:56:26.702488 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.702492 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.702497 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.702501 | controller |     - '          exit 1'
2026-03-14 09:56:26.702505 | controller |     - '        fi'
2026-03-14 09:56:26.702509 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.702514 | controller |     - '        '
2026-03-14 09:56:26.702518 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.702522 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.702526 | controller |       | base64 -d)'
2026-03-14 09:56:26.702531 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.702535 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.702539 | controller |     - '          exit 1'
2026-03-14 09:56:26.702544 | controller |     - '        fi'
2026-03-14 09:56:26.702548 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.702552 | controller |     - '        '
2026-03-14 09:56:26.702557 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.702561 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.702565 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.702570 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.702576 | controller |       --timeout=300s'
2026-03-14 09:56:26.702582 | controller |     - '        '
2026-03-14 09:56:26.702587 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.702592 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.702609 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.702614 | controller |     - '        '
2026-03-14 09:56:26.702618 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.702629 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.702634 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.702638 | controller |     - '        '
2026-03-14 09:56:26.702642 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.702647 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.702651 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.702655 | controller |       available"'
2026-03-14 09:56:26.702660 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.702664 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.702668 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.702672 | controller |     - '          exit 1'
2026-03-14 09:56:26.702677 | controller |     - '        fi'
2026-03-14 09:56:26.702681 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.702685 | controller |     - '        '
2026-03-14 09:56:26.702690 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.702694 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.702698 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.702702 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.702707 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.702711 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.702718 | controller |       -- \'
2026-03-14 09:56:26.702722 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.702726 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.702730 | controller |     - '    logger.go:42: 09:41:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.702735 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.702739 | controller |     - '    logger.go:42: 09:41:51 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.702743 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.702748 | controller |       found'
2026-03-14 09:56:26.702752 | controller |     - '    logger.go:42: 09:41:52 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.702756 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.702760 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.702765 | controller |     - '        '
2026-03-14 09:56:26.702771 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.702777 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.702782 | controller |     - '        '
2026-03-14 09:56:26.702789 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.702797 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.702802 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.702806 | controller |     - '          exit 1'
2026-03-14 09:56:26.702811 | controller |     - '        fi'
2026-03-14 09:56:26.702815 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.702819 | controller |     - '        '
2026-03-14 09:56:26.702823 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.702828 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.702832 | controller |       | base64 -d)'
2026-03-14 09:56:26.702836 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.702841 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.702845 | controller |     - '          exit 1'
2026-03-14 09:56:26.702849 | controller |     - '        fi'
2026-03-14 09:56:26.702853 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.702858 | controller |     - '        '
2026-03-14 09:56:26.702862 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.702866 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.702870 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.702875 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.702879 | controller |       --timeout=300s'
2026-03-14 09:56:26.702883 | controller |     - '        '
2026-03-14 09:56:26.702887 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.702892 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.702896 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.702900 | controller |     - '        '
2026-03-14 09:56:26.702905 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.702916 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.702921 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.702925 | controller |     - '        '
2026-03-14 09:56:26.702930 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.702934 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.702938 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.702942 | controller |       available"'
2026-03-14 09:56:26.702947 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.702951 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.702957 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.702962 | controller |     - '          exit 1'
2026-03-14 09:56:26.702966 | controller |     - '        fi'
2026-03-14 09:56:26.702970 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.702974 | controller |     - '        '
2026-03-14 09:56:26.702979 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.702998 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.703003 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.703007 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.703013 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.703018 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.703022 | controller |       -- \'
2026-03-14 09:56:26.703026 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.703030 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.703035 | controller |     - '    logger.go:42: 09:41:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.703039 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.703043 | controller |     - '    logger.go:42: 09:41:52 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.703048 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.703052 | controller |       found'
2026-03-14 09:56:26.703056 | controller |     - '    logger.go:42: 09:41:53 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.703060 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.703065 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.703069 | controller |     - '        '
2026-03-14 09:56:26.703073 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.703078 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.703082 | controller |     - '        '
2026-03-14 09:56:26.703086 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.703090 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.703095 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.703099 | controller |     - '          exit 1'
2026-03-14 09:56:26.703103 | controller |     - '        fi'
2026-03-14 09:56:26.703107 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.703112 | controller |     - '        '
2026-03-14 09:56:26.703116 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.703120 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.703124 | controller |       | base64 -d)'
2026-03-14 09:56:26.703129 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.703133 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.703137 | controller |     - '          exit 1'
2026-03-14 09:56:26.703141 | controller |     - '        fi'
2026-03-14 09:56:26.703146 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.703150 | controller |     - '        '
2026-03-14 09:56:26.703154 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.703159 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.703163 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.703167 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.703171 | controller |       --timeout=300s'
2026-03-14 09:56:26.703176 | controller |     - '        '
2026-03-14 09:56:26.703180 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.703184 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.703188 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.703193 | controller |     - '        '
2026-03-14 09:56:26.703197 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.703203 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.703216 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.703220 | controller |     - '        '
2026-03-14 09:56:26.703225 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.703229 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.703233 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.703238 | controller |       available"'
2026-03-14 09:56:26.703242 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.703246 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.703250 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.703255 | controller |     - '          exit 1'
2026-03-14 09:56:26.703259 | controller |     - '        fi'
2026-03-14 09:56:26.703263 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.703268 | controller |     - '        '
2026-03-14 09:56:26.703272 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.703276 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.703280 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.703285 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.703289 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.703293 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.703297 | controller |       -- \'
2026-03-14 09:56:26.703302 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.703308 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.703312 | controller |     - '    logger.go:42: 09:41:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.703316 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.703321 | controller |     - '    logger.go:42: 09:41:54 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.703325 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.703329 | controller |       found'
2026-03-14 09:56:26.703334 | controller |     - '    logger.go:42: 09:41:55 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.703339 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.703345 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.703349 | controller |     - '        '
2026-03-14 09:56:26.703354 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.703359 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.703364 | controller |     - '        '
2026-03-14 09:56:26.703369 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.703374 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.703380 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.703386 | controller |     - '          exit 1'
2026-03-14 09:56:26.703392 | controller |     - '        fi'
2026-03-14 09:56:26.703398 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.703404 | controller |     - '        '
2026-03-14 09:56:26.703410 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.703414 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.703421 | controller |       | base64 -d)'
2026-03-14 09:56:26.703425 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.703429 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.703434 | controller |     - '          exit 1'
2026-03-14 09:56:26.703438 | controller |     - '        fi'
2026-03-14 09:56:26.703442 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.703446 | controller |     - '        '
2026-03-14 09:56:26.703451 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.703455 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.703459 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.703464 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.703468 | controller |       --timeout=300s'
2026-03-14 09:56:26.703472 | controller |     - '        '
2026-03-14 09:56:26.703477 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.703481 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.703485 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.703489 | controller |     - '        '
2026-03-14 09:56:26.703494 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.703498 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.703510 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.703514 | controller |     - '        '
2026-03-14 09:56:26.703519 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.703523 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.703527 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.703532 | controller |       available"'
2026-03-14 09:56:26.703536 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.703540 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.703544 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.703549 | controller |     - '          exit 1'
2026-03-14 09:56:26.703553 | controller |     - '        fi'
2026-03-14 09:56:26.703557 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.703562 | controller |     - '        '
2026-03-14 09:56:26.703566 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.703570 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.703574 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.703579 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.703583 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.703587 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.703592 | controller |       -- \'
2026-03-14 09:56:26.703607 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.703614 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.703620 | controller |     - '    logger.go:42: 09:41:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.703625 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.703629 | controller |     - '    logger.go:42: 09:41:55 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.703633 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.703637 | controller |       found'
2026-03-14 09:56:26.703642 | controller |     - '    logger.go:42: 09:41:56 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.703646 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.703654 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.703659 | controller |     - '        '
2026-03-14 09:56:26.703663 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.703668 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.703672 | controller |     - '        '
2026-03-14 09:56:26.703676 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.703680 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.703685 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.703689 | controller |     - '          exit 1'
2026-03-14 09:56:26.703693 | controller |     - '        fi'
2026-03-14 09:56:26.703698 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.703702 | controller |     - '        '
2026-03-14 09:56:26.703708 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.703712 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.703716 | controller |       | base64 -d)'
2026-03-14 09:56:26.703721 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.703725 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.703729 | controller |     - '          exit 1'
2026-03-14 09:56:26.703733 | controller |     - '        fi'
2026-03-14 09:56:26.703738 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.703742 | controller |     - '        '
2026-03-14 09:56:26.703746 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.703750 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.703755 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.703759 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.703765 | controller |       --timeout=300s'
2026-03-14 09:56:26.703770 | controller |     - '        '
2026-03-14 09:56:26.703776 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.703782 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.703788 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.703794 | controller |     - '        '
2026-03-14 09:56:26.703800 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.703804 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.703816 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.703821 | controller |     - '        '
2026-03-14 09:56:26.703825 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.703830 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.703834 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.703844 | controller |       available"'
2026-03-14 09:56:26.703848 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.703852 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.703857 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.703861 | controller |     - '          exit 1'
2026-03-14 09:56:26.703865 | controller |     - '        fi'
2026-03-14 09:56:26.703870 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.703874 | controller |     - '        '
2026-03-14 09:56:26.703878 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.703882 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.703887 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.703891 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.703895 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.703899 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.703904 | controller |       -- \'
2026-03-14 09:56:26.703908 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.703912 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.703917 | controller |     - '    logger.go:42: 09:41:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.703921 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.703925 | controller |     - '    logger.go:42: 09:41:56 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.703930 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.703934 | controller |       found'
2026-03-14 09:56:26.703938 | controller |     - '    logger.go:42: 09:41:57 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.703942 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.703947 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.703951 | controller |     - '        '
2026-03-14 09:56:26.703955 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.703960 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.703964 | controller |     - '        '
2026-03-14 09:56:26.703968 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.703972 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.703977 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.703981 | controller |     - '          exit 1'
2026-03-14 09:56:26.703985 | controller |     - '        fi'
2026-03-14 09:56:26.703990 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.703994 | controller |     - '        '
2026-03-14 09:56:26.703998 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.704004 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.704009 | controller |       | base64 -d)'
2026-03-14 09:56:26.704014 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.704019 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.704023 | controller |     - '          exit 1'
2026-03-14 09:56:26.704027 | controller |     - '        fi'
2026-03-14 09:56:26.704032 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.704038 | controller |     - '        '
2026-03-14 09:56:26.704044 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.704052 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.704058 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.704063 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.704068 | controller |       --timeout=300s'
2026-03-14 09:56:26.704074 | controller |     - '        '
2026-03-14 09:56:26.704079 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.704087 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.704092 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.704097 | controller |     - '        '
2026-03-14 09:56:26.704102 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.704108 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.704113 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.704119 | controller |     - '        '
2026-03-14 09:56:26.704134 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.704140 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.704145 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.704151 | controller |       available"'
2026-03-14 09:56:26.704157 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.704163 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.704168 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.704174 | controller |     - '          exit 1'
2026-03-14 09:56:26.704179 | controller |     - '        fi'
2026-03-14 09:56:26.704185 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.704191 | controller |     - '        '
2026-03-14 09:56:26.704197 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.704203 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.704209 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.704214 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.704223 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.704229 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.704235 | controller |       -- \'
2026-03-14 09:56:26.704241 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.704247 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.704252 | controller |     - '    logger.go:42: 09:41:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.704258 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.704264 | controller |     - '    logger.go:42: 09:41:57 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.704269 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.704275 | controller |       found'
2026-03-14 09:56:26.704281 | controller |     - '    logger.go:42: 09:41:58 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.704289 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.704295 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.704301 | controller |     - '        '
2026-03-14 09:56:26.704306 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.704312 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.704318 | controller |     - '        '
2026-03-14 09:56:26.704323 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.704329 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.704335 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.704341 | controller |     - '          exit 1'
2026-03-14 09:56:26.704347 | controller |     - '        fi'
2026-03-14 09:56:26.704353 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.704359 | controller |     - '        '
2026-03-14 09:56:26.704365 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.704375 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.704381 | controller |       | base64 -d)'
2026-03-14 09:56:26.704387 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.704393 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.704400 | controller |     - '          exit 1'
2026-03-14 09:56:26.704406 | controller |     - '        fi'
2026-03-14 09:56:26.704412 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.704418 | controller |     - '        '
2026-03-14 09:56:26.704424 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.704430 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.704436 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.704441 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.704446 | controller |       --timeout=300s'
2026-03-14 09:56:26.704450 | controller |     - '        '
2026-03-14 09:56:26.704456 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.704462 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.704468 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.704475 | controller |     - '        '
2026-03-14 09:56:26.704481 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.704487 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.704493 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.704502 | controller |     - '        '
2026-03-14 09:56:26.704517 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.704522 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.704527 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.704531 | controller |       available"'
2026-03-14 09:56:26.704536 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.704540 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.704544 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.704549 | controller |     - '          exit 1'
2026-03-14 09:56:26.704553 | controller |     - '        fi'
2026-03-14 09:56:26.704558 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.704566 | controller |     - '        '
2026-03-14 09:56:26.704571 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.704575 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.704579 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.704584 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.704588 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.704592 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.704614 | controller |       -- \'
2026-03-14 09:56:26.704618 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.704623 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.704627 | controller |     - '    logger.go:42: 09:41:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.704631 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.704636 | controller |     - '    logger.go:42: 09:41:58 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.704640 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.704645 | controller |       found'
2026-03-14 09:56:26.704649 | controller |     - '    logger.go:42: 09:41:59 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.704653 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.704658 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.704662 | controller |     - '        '
2026-03-14 09:56:26.704667 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.704671 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.704675 | controller |     - '        '
2026-03-14 09:56:26.704680 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.704684 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.704688 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.704693 | controller |     - '          exit 1'
2026-03-14 09:56:26.704697 | controller |     - '        fi'
2026-03-14 09:56:26.704701 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.704706 | controller |     - '        '
2026-03-14 09:56:26.704710 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.704714 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.704719 | controller |       | base64 -d)'
2026-03-14 09:56:26.704723 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.704727 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.704732 | controller |     - '          exit 1'
2026-03-14 09:56:26.704736 | controller |     - '        fi'
2026-03-14 09:56:26.704740 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.704745 | controller |     - '        '
2026-03-14 09:56:26.704749 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.704754 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.704758 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.704763 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.704769 | controller |       --timeout=300s'
2026-03-14 09:56:26.704775 | controller |     - '        '
2026-03-14 09:56:26.704781 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.704790 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.704797 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.704802 | controller |     - '        '
2026-03-14 09:56:26.704806 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.704811 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.704815 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.704819 | controller |     - '        '
2026-03-14 09:56:26.704824 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.704837 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.704841 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.704846 | controller |       available"'
2026-03-14 09:56:26.704850 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.704855 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.704859 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.704864 | controller |     - '          exit 1'
2026-03-14 09:56:26.704868 | controller |     - '        fi'
2026-03-14 09:56:26.704872 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.704877 | controller |     - '        '
2026-03-14 09:56:26.704884 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.704888 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.704893 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.704897 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.704901 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.704906 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.704910 | controller |       -- \'
2026-03-14 09:56:26.704914 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.704919 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.704923 | controller |     - '    logger.go:42: 09:41:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.704927 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.704932 | controller |     - '    logger.go:42: 09:41:59 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.704936 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.704941 | controller |       found'
2026-03-14 09:56:26.704945 | controller |     - '    logger.go:42: 09:42:00 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.704949 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.704954 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.704958 | controller |     - '        '
2026-03-14 09:56:26.704962 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.704967 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.704971 | controller |     - '        '
2026-03-14 09:56:26.704975 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.704980 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.704986 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.704990 | controller |     - '          exit 1'
2026-03-14 09:56:26.704995 | controller |     - '        fi'
2026-03-14 09:56:26.704999 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.705003 | controller |     - '        '
2026-03-14 09:56:26.705007 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.705012 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.705016 | controller |       | base64 -d)'
2026-03-14 09:56:26.705020 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.705072 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.705078 | controller |     - '          exit 1'
2026-03-14 09:56:26.705082 | controller |     - '        fi'
2026-03-14 09:56:26.705086 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.705091 | controller |     - '        '
2026-03-14 09:56:26.705095 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.705099 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.705104 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.705108 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.705112 | controller |       --timeout=300s'
2026-03-14 09:56:26.705116 | controller |     - '        '
2026-03-14 09:56:26.705121 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.705125 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.705129 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.705134 | controller |     - '        '
2026-03-14 09:56:26.705138 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.705142 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.705146 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.705151 | controller |     - '        '
2026-03-14 09:56:26.705155 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.705159 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.705171 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.705175 | controller |       available"'
2026-03-14 09:56:26.705180 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.705184 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.705189 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.705193 | controller |     - '          exit 1'
2026-03-14 09:56:26.705197 | controller |     - '        fi'
2026-03-14 09:56:26.705202 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.705206 | controller |     - '        '
2026-03-14 09:56:26.705210 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.705220 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.705224 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.705229 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.705233 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.705239 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.705244 | controller |       -- \'
2026-03-14 09:56:26.705248 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.705253 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.705257 | controller |     - '    logger.go:42: 09:42:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.705261 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.705266 | controller |     - '    logger.go:42: 09:42:01 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.705270 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.705274 | controller |       found'
2026-03-14 09:56:26.705279 | controller |     - '    logger.go:42: 09:42:02 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.705283 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.705287 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.705292 | controller |     - '        '
2026-03-14 09:56:26.705296 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.705300 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.705305 | controller |     - '        '
2026-03-14 09:56:26.705309 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.705313 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.705318 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.705322 | controller |     - '          exit 1'
2026-03-14 09:56:26.705326 | controller |     - '        fi'
2026-03-14 09:56:26.705331 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.705335 | controller |     - '        '
2026-03-14 09:56:26.705340 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.705344 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.705348 | controller |       | base64 -d)'
2026-03-14 09:56:26.705353 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.705357 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.705361 | controller |     - '          exit 1'
2026-03-14 09:56:26.705366 | controller |     - '        fi'
2026-03-14 09:56:26.705370 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.705374 | controller |     - '        '
2026-03-14 09:56:26.705379 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.705383 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.705387 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.705392 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.705396 | controller |       --timeout=300s'
2026-03-14 09:56:26.705400 | controller |     - '        '
2026-03-14 09:56:26.705405 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.705409 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.705413 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.705418 | controller |     - '        '
2026-03-14 09:56:26.705422 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.705426 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.705433 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.705438 | controller |     - '        '
2026-03-14 09:56:26.705442 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.705446 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.705457 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.705462 | controller |       available"'
2026-03-14 09:56:26.705466 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.705471 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.705475 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.705480 | controller |     - '          exit 1'
2026-03-14 09:56:26.705484 | controller |     - '        fi'
2026-03-14 09:56:26.705488 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.705493 | controller |     - '        '
2026-03-14 09:56:26.705497 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.705501 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.705506 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.705510 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.705515 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.705519 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.705523 | controller |       -- \'
2026-03-14 09:56:26.705528 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.705532 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.705536 | controller |     - '    logger.go:42: 09:42:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.705541 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.705546 | controller |     - '    logger.go:42: 09:42:02 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.705553 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.705559 | controller |       found'
2026-03-14 09:56:26.705565 | controller |     - '    logger.go:42: 09:42:03 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.705571 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.705577 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.705584 | controller |     - '        '
2026-03-14 09:56:26.705589 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.705605 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.705612 | controller |     - '        '
2026-03-14 09:56:26.705617 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.705622 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.705626 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.705630 | controller |     - '          exit 1'
2026-03-14 09:56:26.705635 | controller |     - '        fi'
2026-03-14 09:56:26.705639 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.705648 | controller |     - '        '
2026-03-14 09:56:26.705652 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.705657 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.705665 | controller |       | base64 -d)'
2026-03-14 09:56:26.705669 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.705674 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.705678 | controller |     - '          exit 1'
2026-03-14 09:56:26.705683 | controller |     - '        fi'
2026-03-14 09:56:26.705687 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.705691 | controller |     - '        '
2026-03-14 09:56:26.705696 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.705700 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.705704 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.705709 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.705713 | controller |       --timeout=300s'
2026-03-14 09:56:26.705717 | controller |     - '        '
2026-03-14 09:56:26.705722 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.705726 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.705732 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.705737 | controller |     - '        '
2026-03-14 09:56:26.705741 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.705746 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.705750 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.705754 | controller |     - '        '
2026-03-14 09:56:26.705759 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.705764 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.705770 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.705777 | controller |       available"'
2026-03-14 09:56:26.705793 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.705799 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.705806 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.705810 | controller |     - '          exit 1'
2026-03-14 09:56:26.705815 | controller |     - '        fi'
2026-03-14 09:56:26.705819 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.705823 | controller |     - '        '
2026-03-14 09:56:26.705828 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.705832 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.705837 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.705841 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.705847 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.705851 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.705856 | controller |       -- \'
2026-03-14 09:56:26.705860 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.705865 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.705869 | controller |     - '    logger.go:42: 09:42:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.705875 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.705881 | controller |     - '    logger.go:42: 09:42:03 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.705887 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.705892 | controller |       found'
2026-03-14 09:56:26.705927 | controller |     - '    logger.go:42: 09:42:04 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.705932 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.705936 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.705941 | controller |     - '        '
2026-03-14 09:56:26.705945 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.705949 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.705954 | controller |     - '        '
2026-03-14 09:56:26.705958 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.705962 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.705967 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.705971 | controller |     - '          exit 1'
2026-03-14 09:56:26.705976 | controller |     - '        fi'
2026-03-14 09:56:26.705980 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.705984 | controller |     - '        '
2026-03-14 09:56:26.705989 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.705993 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.705997 | controller |       | base64 -d)'
2026-03-14 09:56:26.706002 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.706006 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.706010 | controller |     - '          exit 1'
2026-03-14 09:56:26.706015 | controller |     - '        fi'
2026-03-14 09:56:26.706019 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.706023 | controller |     - '        '
2026-03-14 09:56:26.706028 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.706032 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.706036 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.706041 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.706045 | controller |       --timeout=300s'
2026-03-14 09:56:26.706049 | controller |     - '        '
2026-03-14 09:56:26.706054 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.706058 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.706065 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.706069 | controller |     - '        '
2026-03-14 09:56:26.706074 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.706078 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.706082 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.706087 | controller |     - '        '
2026-03-14 09:56:26.706091 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.706095 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.706100 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.706106 | controller |       available"'
2026-03-14 09:56:26.706111 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.706122 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.706127 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.706131 | controller |     - '          exit 1'
2026-03-14 09:56:26.706136 | controller |     - '        fi'
2026-03-14 09:56:26.706140 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.706145 | controller |     - '        '
2026-03-14 09:56:26.706149 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.706153 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.706158 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.706162 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.706167 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.706171 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.706175 | controller |       -- \'
2026-03-14 09:56:26.706180 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.706184 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.706188 | controller |     - '    logger.go:42: 09:42:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.706193 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.706197 | controller |     - '    logger.go:42: 09:42:04 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.706201 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.706206 | controller |       found'
2026-03-14 09:56:26.706210 | controller |     - '    logger.go:42: 09:42:05 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.706214 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.706219 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.706223 | controller |     - '        '
2026-03-14 09:56:26.706227 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.706232 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.706236 | controller |     - '        '
2026-03-14 09:56:26.706240 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.706244 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.706249 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.706253 | controller |     - '          exit 1'
2026-03-14 09:56:26.706258 | controller |     - '        fi'
2026-03-14 09:56:26.706262 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.706266 | controller |     - '        '
2026-03-14 09:56:26.706271 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.706275 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.706279 | controller |       | base64 -d)'
2026-03-14 09:56:26.706284 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.706288 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.706292 | controller |     - '          exit 1'
2026-03-14 09:56:26.706297 | controller |     - '        fi'
2026-03-14 09:56:26.706301 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.706305 | controller |     - '        '
2026-03-14 09:56:26.706310 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.706316 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.706320 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.706324 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.706329 | controller |       --timeout=300s'
2026-03-14 09:56:26.706333 | controller |     - '        '
2026-03-14 09:56:26.706339 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.706344 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.706349 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.706354 | controller |     - '        '
2026-03-14 09:56:26.706359 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.706364 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.706369 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.706373 | controller |     - '        '
2026-03-14 09:56:26.706377 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.706382 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.706386 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.706390 | controller |       available"'
2026-03-14 09:56:26.706395 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.706405 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.706410 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.706414 | controller |     - '          exit 1'
2026-03-14 09:56:26.706418 | controller |     - '        fi'
2026-03-14 09:56:26.706423 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.706427 | controller |     - '        '
2026-03-14 09:56:26.706432 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.706436 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.706440 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.706445 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.706449 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.706453 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.706458 | controller |       -- \'
2026-03-14 09:56:26.706462 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.706467 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.706473 | controller |     - '    logger.go:42: 09:42:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.706478 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.706482 | controller |     - '    logger.go:42: 09:42:05 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.706486 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.706491 | controller |       found'
2026-03-14 09:56:26.706495 | controller |     - '    logger.go:42: 09:42:06 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.706499 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.706506 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.706510 | controller |     - '        '
2026-03-14 09:56:26.706514 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.706519 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.706523 | controller |     - '        '
2026-03-14 09:56:26.706527 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.706532 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.706536 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.706541 | controller |     - '          exit 1'
2026-03-14 09:56:26.706545 | controller |     - '        fi'
2026-03-14 09:56:26.706549 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.706554 | controller |     - '        '
2026-03-14 09:56:26.706558 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.706562 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.706567 | controller |       | base64 -d)'
2026-03-14 09:56:26.706571 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.706575 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.706580 | controller |     - '          exit 1'
2026-03-14 09:56:26.706584 | controller |     - '        fi'
2026-03-14 09:56:26.706588 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.706603 | controller |     - '        '
2026-03-14 09:56:26.706611 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.706615 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.706620 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.706624 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.706629 | controller |       --timeout=300s'
2026-03-14 09:56:26.706635 | controller |     - '        '
2026-03-14 09:56:26.706642 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.706648 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.706654 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.706660 | controller |     - '        '
2026-03-14 09:56:26.706666 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.706672 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.706677 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.706681 | controller |     - '        '
2026-03-14 09:56:26.706686 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.706690 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.706694 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.706699 | controller |       available"'
2026-03-14 09:56:26.706703 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.706708 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.706720 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.706724 | controller |     - '          exit 1'
2026-03-14 09:56:26.706729 | controller |     - '        fi'
2026-03-14 09:56:26.706733 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.706738 | controller |     - '        '
2026-03-14 09:56:26.706746 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.706751 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.706755 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.706762 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.706769 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.706775 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.706781 | controller |       -- \'
2026-03-14 09:56:26.706788 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.706794 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.706798 | controller |     - '    logger.go:42: 09:42:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.706803 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.706807 | controller |     - '    logger.go:42: 09:42:06 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.706811 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.706816 | controller |       found'
2026-03-14 09:56:26.706820 | controller |     - '    logger.go:42: 09:42:07 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.706824 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.706829 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.706833 | controller |     - '        '
2026-03-14 09:56:26.706837 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.706842 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.706846 | controller |     - '        '
2026-03-14 09:56:26.706850 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.706855 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.706859 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.706864 | controller |     - '          exit 1'
2026-03-14 09:56:26.706868 | controller |     - '        fi'
2026-03-14 09:56:26.706872 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.706876 | controller |     - '        '
2026-03-14 09:56:26.706881 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.706885 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.706889 | controller |       | base64 -d)'
2026-03-14 09:56:26.706894 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.706898 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.706903 | controller |     - '          exit 1'
2026-03-14 09:56:26.706907 | controller |     - '        fi'
2026-03-14 09:56:26.706911 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.706915 | controller |     - '        '
2026-03-14 09:56:26.706920 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.706924 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.706928 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.706933 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.706937 | controller |       --timeout=300s'
2026-03-14 09:56:26.706942 | controller |     - '        '
2026-03-14 09:56:26.706946 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.706950 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.706957 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.706961 | controller |     - '        '
2026-03-14 09:56:26.706966 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.706970 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.706974 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.706979 | controller |     - '        '
2026-03-14 09:56:26.706983 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.707005 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.707012 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.707016 | controller |       available"'
2026-03-14 09:56:26.707021 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.707025 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.707029 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.707040 | controller |     - '          exit 1'
2026-03-14 09:56:26.707045 | controller |     - '        fi'
2026-03-14 09:56:26.707050 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.707054 | controller |     - '        '
2026-03-14 09:56:26.707058 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.707063 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.707067 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.707071 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.707076 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.707080 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.707087 | controller |       -- \'
2026-03-14 09:56:26.707091 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.707096 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.707100 | controller |     - '    logger.go:42: 09:42:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.707105 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.707109 | controller |     - '    logger.go:42: 09:42:07 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.707115 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.707119 | controller |       found'
2026-03-14 09:56:26.707125 | controller |     - '    logger.go:42: 09:42:08 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.707129 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.707134 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.707138 | controller |     - '        '
2026-03-14 09:56:26.707143 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.707147 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.707151 | controller |     - '        '
2026-03-14 09:56:26.707156 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.707160 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.707164 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.707171 | controller |     - '          exit 1'
2026-03-14 09:56:26.707175 | controller |     - '        fi'
2026-03-14 09:56:26.707180 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.707184 | controller |     - '        '
2026-03-14 09:56:26.707188 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.707194 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.707199 | controller |       | base64 -d)'
2026-03-14 09:56:26.707212 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.707216 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.707229 | controller |     - '          exit 1'
2026-03-14 09:56:26.707236 | controller |     - '        fi'
2026-03-14 09:56:26.707241 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.707245 | controller |     - '        '
2026-03-14 09:56:26.707249 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.707254 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.707258 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.707262 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.707267 | controller |       --timeout=300s'
2026-03-14 09:56:26.707271 | controller |     - '        '
2026-03-14 09:56:26.707276 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.707280 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.707284 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.707289 | controller |     - '        '
2026-03-14 09:56:26.707293 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.707297 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.707302 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.707306 | controller |     - '        '
2026-03-14 09:56:26.707310 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.707315 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.707319 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.707323 | controller |       available"'
2026-03-14 09:56:26.707328 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.707332 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.707336 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.707341 | controller |     - '          exit 1'
2026-03-14 09:56:26.707345 | controller |     - '        fi'
2026-03-14 09:56:26.707357 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.707361 | controller |     - '        '
2026-03-14 09:56:26.707366 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.707370 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.707375 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.707379 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.707383 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.707397 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.707402 | controller |       -- \'
2026-03-14 09:56:26.707406 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.707411 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.707415 | controller |     - '    logger.go:42: 09:42:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.707419 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.707424 | controller |     - '    logger.go:42: 09:42:09 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.707428 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.707432 | controller |       found'
2026-03-14 09:56:26.707437 | controller |     - '    logger.go:42: 09:42:10 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.707441 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.707445 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.707450 | controller |     - '        '
2026-03-14 09:56:26.707454 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.707459 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.707463 | controller |     - '        '
2026-03-14 09:56:26.707467 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.707471 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.707478 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.707482 | controller |     - '          exit 1'
2026-03-14 09:56:26.707487 | controller |     - '        fi'
2026-03-14 09:56:26.707491 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.707496 | controller |     - '        '
2026-03-14 09:56:26.707500 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.707504 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.707509 | controller |       | base64 -d)'
2026-03-14 09:56:26.707513 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.707518 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.707522 | controller |     - '          exit 1'
2026-03-14 09:56:26.707526 | controller |     - '        fi'
2026-03-14 09:56:26.707531 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.707535 | controller |     - '        '
2026-03-14 09:56:26.707540 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.707546 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.707551 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.707556 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.707560 | controller |       --timeout=300s'
2026-03-14 09:56:26.707564 | controller |     - '        '
2026-03-14 09:56:26.707569 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.707574 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.707580 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.707584 | controller |     - '        '
2026-03-14 09:56:26.707589 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.707604 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.707614 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.707618 | controller |     - '        '
2026-03-14 09:56:26.707623 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.707627 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.707632 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.707636 | controller |       available"'
2026-03-14 09:56:26.707640 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.707645 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.707650 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.707654 | controller |     - '          exit 1'
2026-03-14 09:56:26.707659 | controller |     - '        fi'
2026-03-14 09:56:26.707669 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.707674 | controller |     - '        '
2026-03-14 09:56:26.707678 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.707683 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.707687 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.707692 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.707696 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.707700 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.707705 | controller |       -- \'
2026-03-14 09:56:26.707709 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.707713 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.707718 | controller |     - '    logger.go:42: 09:42:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.707722 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.707727 | controller |     - '    logger.go:42: 09:42:10 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.707731 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.707739 | controller |       found'
2026-03-14 09:56:26.707745 | controller |     - '    logger.go:42: 09:42:11 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.707751 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.707757 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.707762 | controller |     - '        '
2026-03-14 09:56:26.707768 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.707774 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.707780 | controller |     - '        '
2026-03-14 09:56:26.707786 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.707792 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.707798 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.707812 | controller |     - '          exit 1'
2026-03-14 09:56:26.707818 | controller |     - '        fi'
2026-03-14 09:56:26.707824 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.707830 | controller |     - '        '
2026-03-14 09:56:26.707836 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.707842 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.707848 | controller |       | base64 -d)'
2026-03-14 09:56:26.707853 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.707862 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.707868 | controller |     - '          exit 1'
2026-03-14 09:56:26.707874 | controller |     - '        fi'
2026-03-14 09:56:26.707879 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.707884 | controller |     - '        '
2026-03-14 09:56:26.707888 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.707892 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.707897 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.707909 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.707926 | controller |       --timeout=300s'
2026-03-14 09:56:26.707931 | controller |     - '        '
2026-03-14 09:56:26.707935 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.707940 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.707945 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.707953 | controller |     - '        '
2026-03-14 09:56:26.707958 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.707962 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.707967 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.707971 | controller |     - '        '
2026-03-14 09:56:26.707976 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.707980 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.707985 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.707989 | controller |       available"'
2026-03-14 09:56:26.707994 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.707998 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.708007 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.708019 | controller |     - '          exit 1'
2026-03-14 09:56:26.708023 | controller |     - '        fi'
2026-03-14 09:56:26.708028 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.708032 | controller |     - '        '
2026-03-14 09:56:26.708046 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.708052 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.708056 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.708061 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.708065 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.708070 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.708078 | controller |       -- \'
2026-03-14 09:56:26.708088 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.708093 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.708097 | controller |     - '    logger.go:42: 09:42:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.708103 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.708108 | controller |     - '    logger.go:42: 09:42:11 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.708112 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.708116 | controller |       found'
2026-03-14 09:56:26.708121 | controller |     - '    logger.go:42: 09:42:12 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.708125 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.708129 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.708134 | controller |     - '        '
2026-03-14 09:56:26.708143 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.708147 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.708152 | controller |     - '        '
2026-03-14 09:56:26.708156 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.708160 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.708164 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.708169 | controller |     - '          exit 1'
2026-03-14 09:56:26.708173 | controller |     - '        fi'
2026-03-14 09:56:26.708178 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.708188 | controller |     - '        '
2026-03-14 09:56:26.708193 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.708197 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.708208 | controller |       | base64 -d)'
2026-03-14 09:56:26.708218 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.708223 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.708228 | controller |     - '          exit 1'
2026-03-14 09:56:26.708232 | controller |     - '        fi'
2026-03-14 09:56:26.708236 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.708241 | controller |     - '        '
2026-03-14 09:56:26.708245 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.708249 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.708254 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.708258 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.708262 | controller |       --timeout=300s'
2026-03-14 09:56:26.708266 | controller |     - '        '
2026-03-14 09:56:26.708271 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.708275 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.708279 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.708284 | controller |     - '        '
2026-03-14 09:56:26.708288 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.708292 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.708296 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.708301 | controller |     - '        '
2026-03-14 09:56:26.708305 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.708309 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.708313 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.708322 | controller |       available"'
2026-03-14 09:56:26.708326 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.708333 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.708337 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.708341 | controller |     - '          exit 1'
2026-03-14 09:56:26.708346 | controller |     - '        fi'
2026-03-14 09:56:26.708350 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.708354 | controller |     - '        '
2026-03-14 09:56:26.708367 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.708373 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.708378 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.708383 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.708388 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.708392 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.708396 | controller |       -- \'
2026-03-14 09:56:26.708401 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.708405 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.708409 | controller |     - '    logger.go:42: 09:42:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.708413 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.708418 | controller |     - '    logger.go:42: 09:42:12 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.708422 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.708426 | controller |       found'
2026-03-14 09:56:26.708431 | controller |     - '    logger.go:42: 09:42:13 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.708437 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.708442 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.708446 | controller |     - '        '
2026-03-14 09:56:26.708450 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.708455 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.708459 | controller |     - '        '
2026-03-14 09:56:26.708465 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.708469 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.708474 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.708478 | controller |     - '          exit 1'
2026-03-14 09:56:26.708483 | controller |     - '        fi'
2026-03-14 09:56:26.708487 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.708491 | controller |     - '        '
2026-03-14 09:56:26.708496 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.708502 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.708507 | controller |       | base64 -d)'
2026-03-14 09:56:26.708511 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.708516 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.708520 | controller |     - '          exit 1'
2026-03-14 09:56:26.708525 | controller |     - '        fi'
2026-03-14 09:56:26.708529 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.708533 | controller |     - '        '
2026-03-14 09:56:26.708538 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.708542 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.708549 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.708553 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.708557 | controller |       --timeout=300s'
2026-03-14 09:56:26.708562 | controller |     - '        '
2026-03-14 09:56:26.708566 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.708576 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.708581 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.708585 | controller |     - '        '
2026-03-14 09:56:26.708590 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.708606 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.708614 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.708619 | controller |     - '        '
2026-03-14 09:56:26.708623 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.708628 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.708632 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.708637 | controller |       available"'
2026-03-14 09:56:26.708641 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.708645 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.708652 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.708656 | controller |     - '          exit 1'
2026-03-14 09:56:26.708661 | controller |     - '        fi'
2026-03-14 09:56:26.708665 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.708669 | controller |     - '        '
2026-03-14 09:56:26.708674 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.708685 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.708690 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.708694 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.708698 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.708703 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.708707 | controller |       -- \'
2026-03-14 09:56:26.708712 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.708716 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.708726 | controller |     - '    logger.go:42: 09:42:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.708730 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.708739 | controller |     - '    logger.go:42: 09:42:13 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.708743 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.708748 | controller |       found'
2026-03-14 09:56:26.708752 | controller |     - '    logger.go:42: 09:42:14 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.708756 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.708761 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.708767 | controller |     - '        '
2026-03-14 09:56:26.708776 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.708783 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.708789 | controller |     - '        '
2026-03-14 09:56:26.708795 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.708801 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.708806 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.708811 | controller |     - '          exit 1'
2026-03-14 09:56:26.708815 | controller |     - '        fi'
2026-03-14 09:56:26.708819 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.708824 | controller |     - '        '
2026-03-14 09:56:26.708828 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.708832 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.708837 | controller |       | base64 -d)'
2026-03-14 09:56:26.708841 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.708846 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.708850 | controller |     - '          exit 1'
2026-03-14 09:56:26.708854 | controller |     - '        fi'
2026-03-14 09:56:26.708859 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.708863 | controller |     - '        '
2026-03-14 09:56:26.708867 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.708872 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.708876 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.708881 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.708885 | controller |       --timeout=300s'
2026-03-14 09:56:26.708889 | controller |     - '        '
2026-03-14 09:56:26.708894 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.708898 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.708902 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.708907 | controller |     - '        '
2026-03-14 09:56:26.708911 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.708916 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.708920 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.708924 | controller |     - '        '
2026-03-14 09:56:26.708929 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.708933 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.708937 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.708942 | controller |       available"'
2026-03-14 09:56:26.708946 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.708950 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.708955 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.708959 | controller |     - '          exit 1'
2026-03-14 09:56:26.708963 | controller |     - '        fi'
2026-03-14 09:56:26.708968 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.708972 | controller |     - '        '
2026-03-14 09:56:26.708976 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.708983 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.708996 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.709001 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.709005 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.709013 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.709017 | controller |       -- \'
2026-03-14 09:56:26.709022 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.709026 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.709031 | controller |     - '    logger.go:42: 09:42:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.709035 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.709040 | controller |     - '    logger.go:42: 09:42:14 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.709044 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.709048 | controller |       found'
2026-03-14 09:56:26.709053 | controller |     - '    logger.go:42: 09:42:15 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.709057 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.709061 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.709066 | controller |     - '        '
2026-03-14 09:56:26.709070 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.709076 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.709081 | controller |     - '        '
2026-03-14 09:56:26.709085 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.709089 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.709094 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.709098 | controller |     - '          exit 1'
2026-03-14 09:56:26.709103 | controller |     - '        fi'
2026-03-14 09:56:26.709107 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.709111 | controller |     - '        '
2026-03-14 09:56:26.709116 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.709120 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.709124 | controller |       | base64 -d)'
2026-03-14 09:56:26.709129 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.709133 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.709138 | controller |     - '          exit 1'
2026-03-14 09:56:26.709142 | controller |     - '        fi'
2026-03-14 09:56:26.709146 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.709151 | controller |     - '        '
2026-03-14 09:56:26.709155 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.709159 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.709164 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.709168 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.709172 | controller |       --timeout=300s'
2026-03-14 09:56:26.709177 | controller |     - '        '
2026-03-14 09:56:26.709181 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.709185 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.709191 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.709196 | controller |     - '        '
2026-03-14 09:56:26.709206 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.709211 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.709215 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.709220 | controller |     - '        '
2026-03-14 09:56:26.709224 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.709228 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.709233 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.709237 | controller |       available"'
2026-03-14 09:56:26.709242 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.709246 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.709250 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.709254 | controller |     - '          exit 1'
2026-03-14 09:56:26.709259 | controller |     - '        fi'
2026-03-14 09:56:26.709263 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.709268 | controller |     - '        '
2026-03-14 09:56:26.709272 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.709276 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.709287 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.709292 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.709297 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.709301 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.709305 | controller |       -- \'
2026-03-14 09:56:26.709310 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.709314 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.709318 | controller |     - '    logger.go:42: 09:42:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.709323 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.709327 | controller |     - '    logger.go:42: 09:42:16 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.709332 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.709336 | controller |       found'
2026-03-14 09:56:26.709340 | controller |     - '    logger.go:42: 09:42:17 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.709345 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.709349 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.709353 | controller |     - '        '
2026-03-14 09:56:26.709358 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.709362 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.709366 | controller |     - '        '
2026-03-14 09:56:26.709371 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.709375 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.709380 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.709386 | controller |     - '          exit 1'
2026-03-14 09:56:26.709390 | controller |     - '        fi'
2026-03-14 09:56:26.709394 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.709399 | controller |     - '        '
2026-03-14 09:56:26.709403 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.709407 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.709412 | controller |       | base64 -d)'
2026-03-14 09:56:26.709416 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.709421 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.709425 | controller |     - '          exit 1'
2026-03-14 09:56:26.709429 | controller |     - '        fi'
2026-03-14 09:56:26.709434 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.709440 | controller |     - '        '
2026-03-14 09:56:26.709444 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.709449 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.709453 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.709457 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.709462 | controller |       --timeout=300s'
2026-03-14 09:56:26.709466 | controller |     - '        '
2026-03-14 09:56:26.709470 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.709475 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.709479 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.709484 | controller |     - '        '
2026-03-14 09:56:26.709488 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.709492 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.709497 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.709501 | controller |     - '        '
2026-03-14 09:56:26.709506 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.709510 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.709514 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.709519 | controller |       available"'
2026-03-14 09:56:26.709525 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.709529 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.709533 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.709538 | controller |     - '          exit 1'
2026-03-14 09:56:26.709542 | controller |     - '        fi'
2026-03-14 09:56:26.709547 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.709551 | controller |     - '        '
2026-03-14 09:56:26.709555 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.709560 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.709564 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.709575 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.709579 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.709584 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.709591 | controller |       -- \'
2026-03-14 09:56:26.709611 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.709616 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.709620 | controller |     - '    logger.go:42: 09:42:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.709624 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.709629 | controller |     - '    logger.go:42: 09:42:17 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.709633 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.709638 | controller |       found'
2026-03-14 09:56:26.709642 | controller |     - '    logger.go:42: 09:42:18 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.709646 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.709651 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.709655 | controller |     - '        '
2026-03-14 09:56:26.709659 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.709664 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.709671 | controller |     - '        '
2026-03-14 09:56:26.709682 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.709686 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.709691 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.709695 | controller |     - '          exit 1'
2026-03-14 09:56:26.709699 | controller |     - '        fi'
2026-03-14 09:56:26.709704 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.709708 | controller |     - '        '
2026-03-14 09:56:26.709712 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.709716 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.709721 | controller |       | base64 -d)'
2026-03-14 09:56:26.709725 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.709729 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.709734 | controller |     - '          exit 1'
2026-03-14 09:56:26.709738 | controller |     - '        fi'
2026-03-14 09:56:26.709742 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.709747 | controller |     - '        '
2026-03-14 09:56:26.709751 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.709755 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.709760 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.709766 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.709772 | controller |       --timeout=300s'
2026-03-14 09:56:26.709778 | controller |     - '        '
2026-03-14 09:56:26.709784 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.709791 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.709795 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.709799 | controller |     - '        '
2026-03-14 09:56:26.709804 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.709808 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.709812 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.709819 | controller |     - '        '
2026-03-14 09:56:26.709823 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.709828 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.709832 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.709836 | controller |       available"'
2026-03-14 09:56:26.709841 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.709845 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.709849 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.709853 | controller |     - '          exit 1'
2026-03-14 09:56:26.709858 | controller |     - '        fi'
2026-03-14 09:56:26.709862 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.709867 | controller |     - '        '
2026-03-14 09:56:26.709871 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.709875 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.709879 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.709896 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.709901 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.709905 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.709909 | controller |       -- \'
2026-03-14 09:56:26.709914 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.709918 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.709922 | controller |     - '    logger.go:42: 09:42:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.709927 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.709931 | controller |     - '    logger.go:42: 09:42:18 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.709935 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.709940 | controller |       found'
2026-03-14 09:56:26.709944 | controller |     - '    logger.go:42: 09:42:19 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.709948 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.709953 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.709957 | controller |     - '        '
2026-03-14 09:56:26.709961 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.709966 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.709970 | controller |     - '        '
2026-03-14 09:56:26.709974 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.709978 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.709983 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.709987 | controller |     - '          exit 1'
2026-03-14 09:56:26.709991 | controller |     - '        fi'
2026-03-14 09:56:26.709996 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.710000 | controller |     - '        '
2026-03-14 09:56:26.710004 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.710011 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.710017 | controller |       | base64 -d)'
2026-03-14 09:56:26.710021 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.710027 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.710031 | controller |     - '          exit 1'
2026-03-14 09:56:26.710036 | controller |     - '        fi'
2026-03-14 09:56:26.710040 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.710044 | controller |     - '        '
2026-03-14 09:56:26.710049 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.710053 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.710057 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.710061 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.710066 | controller |       --timeout=300s'
2026-03-14 09:56:26.710070 | controller |     - '        '
2026-03-14 09:56:26.710074 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.710079 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.710083 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.710087 | controller |     - '        '
2026-03-14 09:56:26.710092 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.710096 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.710102 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.710106 | controller |     - '        '
2026-03-14 09:56:26.710111 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.710115 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.710119 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.710123 | controller |       available"'
2026-03-14 09:56:26.710128 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.710132 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.710136 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.710141 | controller |     - '          exit 1'
2026-03-14 09:56:26.710145 | controller |     - '        fi'
2026-03-14 09:56:26.710149 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.710153 | controller |     - '        '
2026-03-14 09:56:26.710158 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.710162 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.710166 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.710171 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.710181 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.710186 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.710190 | controller |       -- \'
2026-03-14 09:56:26.710194 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.710199 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.710203 | controller |     - '    logger.go:42: 09:42:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.710207 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.710213 | controller |     - '    logger.go:42: 09:42:19 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.710217 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.710222 | controller |       found'
2026-03-14 09:56:26.710226 | controller |     - '    logger.go:42: 09:42:20 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.710230 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.710235 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.710239 | controller |     - '        '
2026-03-14 09:56:26.710243 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.710247 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.710252 | controller |     - '        '
2026-03-14 09:56:26.710256 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.710262 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.710266 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.710271 | controller |     - '          exit 1'
2026-03-14 09:56:26.710275 | controller |     - '        fi'
2026-03-14 09:56:26.710279 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.710284 | controller |     - '        '
2026-03-14 09:56:26.710288 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.710292 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.710296 | controller |       | base64 -d)'
2026-03-14 09:56:26.710301 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.710305 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.710309 | controller |     - '          exit 1'
2026-03-14 09:56:26.710313 | controller |     - '        fi'
2026-03-14 09:56:26.710318 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.710322 | controller |     - '        '
2026-03-14 09:56:26.710326 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.710331 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.710335 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.710341 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.710346 | controller |       --timeout=300s'
2026-03-14 09:56:26.710350 | controller |     - '        '
2026-03-14 09:56:26.710355 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.710360 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.710365 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.710370 | controller |     - '        '
2026-03-14 09:56:26.710374 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.710378 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.710383 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.710387 | controller |     - '        '
2026-03-14 09:56:26.710391 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.710395 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.710400 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.710404 | controller |       available"'
2026-03-14 09:56:26.710408 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.710414 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.710418 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.710423 | controller |     - '          exit 1'
2026-03-14 09:56:26.710427 | controller |     - '        fi'
2026-03-14 09:56:26.710431 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.710436 | controller |     - '        '
2026-03-14 09:56:26.710440 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.710444 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.710449 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.710453 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.710463 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.710468 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.710472 | controller |       -- \'
2026-03-14 09:56:26.710476 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.710481 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.710485 | controller |     - '    logger.go:42: 09:42:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.710490 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.710494 | controller |     - '    logger.go:42: 09:42:20 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.710498 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.710503 | controller |       found'
2026-03-14 09:56:26.710507 | controller |     - '    logger.go:42: 09:42:21 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.710511 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.710515 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.710520 | controller |     - '        '
2026-03-14 09:56:26.710524 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.710528 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.710533 | controller |     - '        '
2026-03-14 09:56:26.710537 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.710541 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.710545 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.710550 | controller |     - '          exit 1'
2026-03-14 09:56:26.710554 | controller |     - '        fi'
2026-03-14 09:56:26.710558 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.710563 | controller |     - '        '
2026-03-14 09:56:26.710567 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.710571 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.710575 | controller |       | base64 -d)'
2026-03-14 09:56:26.710580 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.710584 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.710588 | controller |     - '          exit 1'
2026-03-14 09:56:26.710593 | controller |     - '        fi'
2026-03-14 09:56:26.710614 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.710619 | controller |     - '        '
2026-03-14 09:56:26.710625 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.710630 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.710634 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.710640 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.710646 | controller |       --timeout=300s'
2026-03-14 09:56:26.710651 | controller |     - '        '
2026-03-14 09:56:26.710656 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.710661 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.710665 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.710669 | controller |     - '        '
2026-03-14 09:56:26.710674 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.710678 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.710682 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.710687 | controller |     - '        '
2026-03-14 09:56:26.710691 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.710695 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.710700 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.710704 | controller |       available"'
2026-03-14 09:56:26.710708 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.710712 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.710717 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.710721 | controller |     - '          exit 1'
2026-03-14 09:56:26.710725 | controller |     - '        fi'
2026-03-14 09:56:26.710730 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.710734 | controller |     - '        '
2026-03-14 09:56:26.710738 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.710742 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.710747 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.710751 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.710755 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.710768 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.710775 | controller |       -- \'
2026-03-14 09:56:26.710781 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.710788 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.710793 | controller |     - '    logger.go:42: 09:42:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.710798 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.710802 | controller |     - '    logger.go:42: 09:42:21 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.710806 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.710811 | controller |       found'
2026-03-14 09:56:26.710815 | controller |     - '    logger.go:42: 09:42:22 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.710819 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.710823 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.710828 | controller |     - '        '
2026-03-14 09:56:26.710832 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.710838 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.710843 | controller |     - '        '
2026-03-14 09:56:26.710847 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.710851 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.710856 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.710860 | controller |     - '          exit 1'
2026-03-14 09:56:26.710864 | controller |     - '        fi'
2026-03-14 09:56:26.710869 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.710873 | controller |     - '        '
2026-03-14 09:56:26.710877 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.710882 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.710886 | controller |       | base64 -d)'
2026-03-14 09:56:26.710890 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.710894 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.710899 | controller |     - '          exit 1'
2026-03-14 09:56:26.710903 | controller |     - '        fi'
2026-03-14 09:56:26.710907 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.710912 | controller |     - '        '
2026-03-14 09:56:26.710916 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.710920 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.710927 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.710931 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.710936 | controller |       --timeout=300s'
2026-03-14 09:56:26.710940 | controller |     - '        '
2026-03-14 09:56:26.710944 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.710949 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.710953 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.710957 | controller |     - '        '
2026-03-14 09:56:26.710962 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.710966 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.710970 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.710975 | controller |     - '        '
2026-03-14 09:56:26.710979 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.710983 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.711005 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.711010 | controller |       available"'
2026-03-14 09:56:26.711014 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.711019 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.711023 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.711027 | controller |     - '          exit 1'
2026-03-14 09:56:26.711031 | controller |     - '        fi'
2026-03-14 09:56:26.711036 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.711040 | controller |     - '        '
2026-03-14 09:56:26.711044 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.711049 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.711056 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.711061 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.711065 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.711077 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.711082 | controller |       -- \'
2026-03-14 09:56:26.711086 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.711092 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.711097 | controller |     - '    logger.go:42: 09:42:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.711101 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.711105 | controller |     - '    logger.go:42: 09:42:23 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.711110 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.711114 | controller |       found'
2026-03-14 09:56:26.711118 | controller |     - '    logger.go:42: 09:42:24 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.711122 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.711127 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.711131 | controller |     - '        '
2026-03-14 09:56:26.711135 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.711140 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.711144 | controller |     - '        '
2026-03-14 09:56:26.711148 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.711152 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.711157 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.711161 | controller |     - '          exit 1'
2026-03-14 09:56:26.711165 | controller |     - '        fi'
2026-03-14 09:56:26.711170 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.711174 | controller |     - '        '
2026-03-14 09:56:26.711178 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.711182 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.711187 | controller |       | base64 -d)'
2026-03-14 09:56:26.711191 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.711195 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.711199 | controller |     - '          exit 1'
2026-03-14 09:56:26.711204 | controller |     - '        fi'
2026-03-14 09:56:26.711208 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.711212 | controller |     - '        '
2026-03-14 09:56:26.711216 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.711221 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.711225 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.711229 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.711233 | controller |       --timeout=300s'
2026-03-14 09:56:26.711238 | controller |     - '        '
2026-03-14 09:56:26.711242 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.711246 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.711251 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.711257 | controller |     - '        '
2026-03-14 09:56:26.711261 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.711265 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.711269 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.711274 | controller |     - '        '
2026-03-14 09:56:26.711278 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.711282 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.711287 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.711291 | controller |       available"'
2026-03-14 09:56:26.711295 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.711300 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.711306 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.711310 | controller |     - '          exit 1'
2026-03-14 09:56:26.711314 | controller |     - '        fi'
2026-03-14 09:56:26.711319 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.711323 | controller |     - '        '
2026-03-14 09:56:26.711327 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.711332 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.711336 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.711340 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.711345 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.711349 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.711355 | controller |       -- \'
2026-03-14 09:56:26.711365 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.711370 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.711374 | controller |     - '    logger.go:42: 09:42:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.711379 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.711383 | controller |     - '    logger.go:42: 09:42:24 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.711387 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.711392 | controller |       found'
2026-03-14 09:56:26.711396 | controller |     - '    logger.go:42: 09:42:25 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.711400 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.711406 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.711411 | controller |     - '        '
2026-03-14 09:56:26.711415 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.711420 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.711424 | controller |     - '        '
2026-03-14 09:56:26.711428 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.711433 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.711437 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.711442 | controller |     - '          exit 1'
2026-03-14 09:56:26.711446 | controller |     - '        fi'
2026-03-14 09:56:26.711453 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.711457 | controller |     - '        '
2026-03-14 09:56:26.711461 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.711465 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.711470 | controller |       | base64 -d)'
2026-03-14 09:56:26.711474 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.711478 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.711483 | controller |     - '          exit 1'
2026-03-14 09:56:26.711487 | controller |     - '        fi'
2026-03-14 09:56:26.711491 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.711496 | controller |     - '        '
2026-03-14 09:56:26.711500 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.711504 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.711508 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.711514 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.711519 | controller |       --timeout=300s'
2026-03-14 09:56:26.711523 | controller |     - '        '
2026-03-14 09:56:26.711527 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.711532 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.711536 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.711540 | controller |     - '        '
2026-03-14 09:56:26.711544 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.711549 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.711553 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.711557 | controller |     - '        '
2026-03-14 09:56:26.711562 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.711566 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.711570 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.711574 | controller |       available"'
2026-03-14 09:56:26.711579 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.711583 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.711587 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.711592 | controller |     - '          exit 1'
2026-03-14 09:56:26.711607 | controller |     - '        fi'
2026-03-14 09:56:26.711614 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.711618 | controller |     - '        '
2026-03-14 09:56:26.711623 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.711627 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.711631 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.711635 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.711640 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.711644 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.711650 | controller |       -- \'
2026-03-14 09:56:26.711662 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.711667 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.711671 | controller |     - '    logger.go:42: 09:42:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.711675 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.711680 | controller |     - '    logger.go:42: 09:42:25 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.711684 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.711688 | controller |       found'
2026-03-14 09:56:26.711693 | controller |     - '    logger.go:42: 09:42:26 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.711697 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.711701 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.711705 | controller |     - '        '
2026-03-14 09:56:26.711710 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.711714 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.711718 | controller |     - '        '
2026-03-14 09:56:26.711723 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.711727 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.711731 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.711736 | controller |     - '          exit 1'
2026-03-14 09:56:26.711740 | controller |     - '        fi'
2026-03-14 09:56:26.711747 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.711751 | controller |     - '        '
2026-03-14 09:56:26.711755 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.711759 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.711765 | controller |       | base64 -d)'
2026-03-14 09:56:26.711771 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.711777 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.711784 | controller |     - '          exit 1'
2026-03-14 09:56:26.711790 | controller |     - '        fi'
2026-03-14 09:56:26.711795 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.711799 | controller |     - '        '
2026-03-14 09:56:26.711804 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.711808 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.711812 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.711816 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.711821 | controller |       --timeout=300s'
2026-03-14 09:56:26.711825 | controller |     - '        '
2026-03-14 09:56:26.711829 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.711834 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.711838 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.711842 | controller |     - '        '
2026-03-14 09:56:26.711846 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.711851 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.711855 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.711861 | controller |     - '        '
2026-03-14 09:56:26.711866 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.711872 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.711877 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.711881 | controller |       available"'
2026-03-14 09:56:26.711885 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.711889 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.711894 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.711898 | controller |     - '          exit 1'
2026-03-14 09:56:26.711902 | controller |     - '        fi'
2026-03-14 09:56:26.711907 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.711911 | controller |     - '        '
2026-03-14 09:56:26.711915 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.711919 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.711924 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.711928 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.711932 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.711936 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.711941 | controller |       -- \'
2026-03-14 09:56:26.711953 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.711957 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.711962 | controller |     - '    logger.go:42: 09:42:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.711966 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.711970 | controller |     - '    logger.go:42: 09:42:26 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.711975 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.711979 | controller |       found'
2026-03-14 09:56:26.711983 | controller |     - '    logger.go:42: 09:42:27 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.711987 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.711992 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.711996 | controller |     - '        '
2026-03-14 09:56:26.712000 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.712005 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.712009 | controller |     - '        '
2026-03-14 09:56:26.712013 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.712017 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.712022 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.712026 | controller |     - '          exit 1'
2026-03-14 09:56:26.712030 | controller |     - '        fi'
2026-03-14 09:56:26.712034 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.712039 | controller |     - '        '
2026-03-14 09:56:26.712043 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.712047 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.712052 | controller |       | base64 -d)'
2026-03-14 09:56:26.712056 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.712060 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.712067 | controller |     - '          exit 1'
2026-03-14 09:56:26.712072 | controller |     - '        fi'
2026-03-14 09:56:26.712076 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.712080 | controller |     - '        '
2026-03-14 09:56:26.712084 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.712089 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.712093 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.712097 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.712104 | controller |       --timeout=300s'
2026-03-14 09:56:26.712108 | controller |     - '        '
2026-03-14 09:56:26.712112 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.712116 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.712121 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.712125 | controller |     - '        '
2026-03-14 09:56:26.712129 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.712134 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.712138 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.712144 | controller |     - '        '
2026-03-14 09:56:26.712149 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.712153 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.712157 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.712161 | controller |       available"'
2026-03-14 09:56:26.712166 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.712170 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.712174 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.712178 | controller |     - '          exit 1'
2026-03-14 09:56:26.712183 | controller |     - '        fi'
2026-03-14 09:56:26.712187 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.712191 | controller |     - '        '
2026-03-14 09:56:26.712196 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.712200 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.712204 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.712208 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.712213 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.712217 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.712221 | controller |       -- \'
2026-03-14 09:56:26.712226 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.712236 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.712241 | controller |     - '    logger.go:42: 09:42:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.712245 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.712249 | controller |     - '    logger.go:42: 09:42:27 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.712257 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.712261 | controller |       found'
2026-03-14 09:56:26.712266 | controller |     - '    logger.go:42: 09:42:28 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.712270 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.712274 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.712278 | controller |     - '        '
2026-03-14 09:56:26.712283 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.712287 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.712291 | controller |     - '        '
2026-03-14 09:56:26.712296 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.712300 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.712304 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.712309 | controller |     - '          exit 1'
2026-03-14 09:56:26.712313 | controller |     - '        fi'
2026-03-14 09:56:26.712317 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.712322 | controller |     - '        '
2026-03-14 09:56:26.712326 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.712330 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.712334 | controller |       | base64 -d)'
2026-03-14 09:56:26.712339 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.712343 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.712347 | controller |     - '          exit 1'
2026-03-14 09:56:26.712352 | controller |     - '        fi'
2026-03-14 09:56:26.712356 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.712360 | controller |     - '        '
2026-03-14 09:56:26.712365 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.712371 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.712376 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.712380 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.712385 | controller |       --timeout=300s'
2026-03-14 09:56:26.712389 | controller |     - '        '
2026-03-14 09:56:26.712393 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.712398 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.712402 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.712406 | controller |     - '        '
2026-03-14 09:56:26.712410 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.712415 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.712419 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.712423 | controller |     - '        '
2026-03-14 09:56:26.712428 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.712432 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.712436 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.712440 | controller |       available"'
2026-03-14 09:56:26.712445 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.712449 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.712456 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.712461 | controller |     - '          exit 1'
2026-03-14 09:56:26.712466 | controller |     - '        fi'
2026-03-14 09:56:26.712470 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.712474 | controller |     - '        '
2026-03-14 09:56:26.712479 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.712483 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.712487 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.712491 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.712496 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.712500 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.712504 | controller |       -- \'
2026-03-14 09:56:26.712509 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.712513 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.712524 | controller |     - '    logger.go:42: 09:42:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.712529 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.712533 | controller |     - '    logger.go:42: 09:42:28 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.712537 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.712542 | controller |       found'
2026-03-14 09:56:26.712546 | controller |     - '    logger.go:42: 09:42:29 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.712550 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.712554 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.712559 | controller |     - '        '
2026-03-14 09:56:26.712563 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.712567 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.712571 | controller |     - '        '
2026-03-14 09:56:26.712576 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.712580 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.712584 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.712589 | controller |     - '          exit 1'
2026-03-14 09:56:26.712605 | controller |     - '        fi'
2026-03-14 09:56:26.712611 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.712616 | controller |     - '        '
2026-03-14 09:56:26.712620 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.712626 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.712631 | controller |       | base64 -d)'
2026-03-14 09:56:26.712636 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.712642 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.712646 | controller |     - '          exit 1'
2026-03-14 09:56:26.712651 | controller |     - '        fi'
2026-03-14 09:56:26.712655 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.712659 | controller |     - '        '
2026-03-14 09:56:26.712664 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.712668 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.712672 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.712678 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.712682 | controller |       --timeout=300s'
2026-03-14 09:56:26.712687 | controller |     - '        '
2026-03-14 09:56:26.712691 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.712697 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.712701 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.712706 | controller |     - '        '
2026-03-14 09:56:26.712710 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.712714 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.712719 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.712723 | controller |     - '        '
2026-03-14 09:56:26.712727 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.712733 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.712737 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.712742 | controller |       available"'
2026-03-14 09:56:26.712746 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.712750 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.712755 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.712759 | controller |     - '          exit 1'
2026-03-14 09:56:26.712765 | controller |     - '        fi'
2026-03-14 09:56:26.712773 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.712779 | controller |     - '        '
2026-03-14 09:56:26.712786 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.712792 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.712796 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.712800 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.712805 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.712809 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.712814 | controller |       -- \'
2026-03-14 09:56:26.712818 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.712822 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.712835 | controller |     - '    logger.go:42: 09:42:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.712839 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.712844 | controller |     - '    logger.go:42: 09:42:30 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.712848 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.712852 | controller |       found'
2026-03-14 09:56:26.712857 | controller |     - '    logger.go:42: 09:42:31 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.712861 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.712865 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.712870 | controller |     - '        '
2026-03-14 09:56:26.712874 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.712880 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.712885 | controller |     - '        '
2026-03-14 09:56:26.712889 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.712893 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.712897 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.712902 | controller |     - '          exit 1'
2026-03-14 09:56:26.712906 | controller |     - '        fi'
2026-03-14 09:56:26.712910 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.712914 | controller |     - '        '
2026-03-14 09:56:26.712919 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.712923 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.712927 | controller |       | base64 -d)'
2026-03-14 09:56:26.712932 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.712936 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.712940 | controller |     - '          exit 1'
2026-03-14 09:56:26.712944 | controller |     - '        fi'
2026-03-14 09:56:26.712949 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.712953 | controller |     - '        '
2026-03-14 09:56:26.712957 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.712962 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.712966 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.712970 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.712974 | controller |       --timeout=300s'
2026-03-14 09:56:26.712979 | controller |     - '        '
2026-03-14 09:56:26.712983 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.712987 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.712992 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.712996 | controller |     - '        '
2026-03-14 09:56:26.713000 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.713004 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.713009 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.713013 | controller |     - '        '
2026-03-14 09:56:26.713017 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.713022 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.713026 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.713030 | controller |       available"'
2026-03-14 09:56:26.713034 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.713039 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.713043 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.713047 | controller |     - '          exit 1'
2026-03-14 09:56:26.713051 | controller |     - '        fi'
2026-03-14 09:56:26.713056 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.713060 | controller |     - '        '
2026-03-14 09:56:26.713064 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.713068 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.713074 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.713079 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.713083 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.713087 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.713092 | controller |       -- \'
2026-03-14 09:56:26.713096 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.713100 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.713110 | controller |     - '    logger.go:42: 09:42:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.713115 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.713119 | controller |     - '    logger.go:42: 09:42:31 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.713124 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.713128 | controller |       found'
2026-03-14 09:56:26.713132 | controller |     - '    logger.go:42: 09:42:32 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.713137 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.713141 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.713145 | controller |     - '        '
2026-03-14 09:56:26.713150 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.713154 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.713158 | controller |     - '        '
2026-03-14 09:56:26.713163 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.713167 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.713171 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.713176 | controller |     - '          exit 1'
2026-03-14 09:56:26.713180 | controller |     - '        fi'
2026-03-14 09:56:26.713184 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.713189 | controller |     - '        '
2026-03-14 09:56:26.713193 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.713197 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.713201 | controller |       | base64 -d)'
2026-03-14 09:56:26.713206 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.713210 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.713214 | controller |     - '          exit 1'
2026-03-14 09:56:26.713219 | controller |     - '        fi'
2026-03-14 09:56:26.713223 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.713227 | controller |     - '        '
2026-03-14 09:56:26.713232 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.713236 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.713240 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.713245 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.713249 | controller |       --timeout=300s'
2026-03-14 09:56:26.713253 | controller |     - '        '
2026-03-14 09:56:26.713257 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.713262 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.713268 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.713274 | controller |     - '        '
2026-03-14 09:56:26.713279 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.713283 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.713287 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.713292 | controller |     - '        '
2026-03-14 09:56:26.713296 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.713300 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.713305 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.713319 | controller |       available"'
2026-03-14 09:56:26.713323 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.713327 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.713332 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.713336 | controller |     - '          exit 1'
2026-03-14 09:56:26.713340 | controller |     - '        fi'
2026-03-14 09:56:26.713345 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.713349 | controller |     - '        '
2026-03-14 09:56:26.713354 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.713359 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.713365 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.713369 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.713374 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.713380 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.713384 | controller |       -- \'
2026-03-14 09:56:26.713389 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.713393 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.713397 | controller |     - '    logger.go:42: 09:42:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.713402 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.713412 | controller |     - '    logger.go:42: 09:42:32 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.713417 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.713421 | controller |       found'
2026-03-14 09:56:26.713425 | controller |     - '    logger.go:42: 09:42:33 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.713430 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.713434 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.713438 | controller |     - '        '
2026-03-14 09:56:26.713443 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.713447 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.713451 | controller |     - '        '
2026-03-14 09:56:26.713455 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.713460 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.713464 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.713468 | controller |     - '          exit 1'
2026-03-14 09:56:26.713473 | controller |     - '        fi'
2026-03-14 09:56:26.713477 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.713483 | controller |     - '        '
2026-03-14 09:56:26.713487 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.713492 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.713496 | controller |       | base64 -d)'
2026-03-14 09:56:26.713500 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.713505 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.713509 | controller |     - '          exit 1'
2026-03-14 09:56:26.713513 | controller |     - '        fi'
2026-03-14 09:56:26.713517 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.713522 | controller |     - '        '
2026-03-14 09:56:26.713526 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.713530 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.713535 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.713539 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.713543 | controller |       --timeout=300s'
2026-03-14 09:56:26.713547 | controller |     - '        '
2026-03-14 09:56:26.713552 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.713556 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.713560 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.713565 | controller |     - '        '
2026-03-14 09:56:26.713569 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.713573 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.713577 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.713582 | controller |     - '        '
2026-03-14 09:56:26.713586 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.713590 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.713618 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.713625 | controller |       available"'
2026-03-14 09:56:26.713630 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.713634 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.713638 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.713642 | controller |     - '          exit 1'
2026-03-14 09:56:26.713647 | controller |     - '        fi'
2026-03-14 09:56:26.713651 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.713655 | controller |     - '        '
2026-03-14 09:56:26.713660 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.713664 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.713668 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.713672 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.713677 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.713681 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.713685 | controller |       -- \'
2026-03-14 09:56:26.713722 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.713727 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.713731 | controller |     - '    logger.go:42: 09:42:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.713735 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.713747 | controller |     - '    logger.go:42: 09:42:33 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.713751 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.713756 | controller |       found'
2026-03-14 09:56:26.713760 | controller |     - '    logger.go:42: 09:42:34 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.713766 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.713772 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.713779 | controller |     - '        '
2026-03-14 09:56:26.713785 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.713791 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.713795 | controller |     - '        '
2026-03-14 09:56:26.713800 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.713804 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.713808 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.713812 | controller |     - '          exit 1'
2026-03-14 09:56:26.713817 | controller |     - '        fi'
2026-03-14 09:56:26.713821 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.713825 | controller |     - '        '
2026-03-14 09:56:26.713830 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.713834 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.713838 | controller |       | base64 -d)'
2026-03-14 09:56:26.713842 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.713847 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.713851 | controller |     - '          exit 1'
2026-03-14 09:56:26.713855 | controller |     - '        fi'
2026-03-14 09:56:26.713860 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.713864 | controller |     - '        '
2026-03-14 09:56:26.713868 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.713872 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.713877 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.713881 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.713885 | controller |       --timeout=300s'
2026-03-14 09:56:26.713892 | controller |     - '        '
2026-03-14 09:56:26.713897 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.713901 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.713905 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.713910 | controller |     - '        '
2026-03-14 09:56:26.713914 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.713918 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.713923 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.713927 | controller |     - '        '
2026-03-14 09:56:26.713931 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.713939 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.713943 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.713947 | controller |       available"'
2026-03-14 09:56:26.713952 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.713956 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.713962 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.713966 | controller |     - '          exit 1'
2026-03-14 09:56:26.713971 | controller |     - '        fi'
2026-03-14 09:56:26.713975 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.713981 | controller |     - '        '
2026-03-14 09:56:26.713985 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.713989 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.713994 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.713998 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.714002 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.714006 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.714011 | controller |       -- \'
2026-03-14 09:56:26.714015 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.714019 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.714024 | controller |     - '    logger.go:42: 09:42:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.714028 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.714032 | controller |     - '    logger.go:42: 09:42:34 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.714044 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.714049 | controller |       found'
2026-03-14 09:56:26.714053 | controller |     - '    logger.go:42: 09:42:35 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.714057 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.714062 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.714066 | controller |     - '        '
2026-03-14 09:56:26.714071 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.714075 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.714079 | controller |     - '        '
2026-03-14 09:56:26.714083 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.714088 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.714092 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.714096 | controller |     - '          exit 1'
2026-03-14 09:56:26.714101 | controller |     - '        fi'
2026-03-14 09:56:26.714105 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.714111 | controller |     - '        '
2026-03-14 09:56:26.714115 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.714120 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.714125 | controller |       | base64 -d)'
2026-03-14 09:56:26.714130 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.714134 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.714138 | controller |     - '          exit 1'
2026-03-14 09:56:26.714144 | controller |     - '        fi'
2026-03-14 09:56:26.714149 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.714153 | controller |     - '        '
2026-03-14 09:56:26.714157 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.714161 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.714166 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.714170 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.714174 | controller |       --timeout=300s'
2026-03-14 09:56:26.714178 | controller |     - '        '
2026-03-14 09:56:26.714183 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.714187 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.714191 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.714195 | controller |     - '        '
2026-03-14 09:56:26.714200 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.714204 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.714208 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.714213 | controller |     - '        '
2026-03-14 09:56:26.714217 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.714221 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.714226 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.714230 | controller |       available"'
2026-03-14 09:56:26.714234 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.714238 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.714243 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.714247 | controller |     - '          exit 1'
2026-03-14 09:56:26.714251 | controller |     - '        fi'
2026-03-14 09:56:26.714255 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.714260 | controller |     - '        '
2026-03-14 09:56:26.714264 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.714268 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.714273 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.714278 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.714283 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.714287 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.714293 | controller |       -- \'
2026-03-14 09:56:26.714297 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.714301 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.714306 | controller |     - '    logger.go:42: 09:42:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.714310 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.714314 | controller |     - '    logger.go:42: 09:42:36 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.714319 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.714331 | controller |       found'
2026-03-14 09:56:26.714353 | controller |     - '    logger.go:42: 09:42:37 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.714357 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.714361 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.714366 | controller |     - '        '
2026-03-14 09:56:26.714370 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.714375 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.714379 | controller |     - '        '
2026-03-14 09:56:26.714383 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.714388 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.714392 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.714396 | controller |     - '          exit 1'
2026-03-14 09:56:26.714401 | controller |     - '        fi'
2026-03-14 09:56:26.714405 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.714409 | controller |     - '        '
2026-03-14 09:56:26.714414 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.714418 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.714422 | controller |       | base64 -d)'
2026-03-14 09:56:26.714427 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.714431 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.714435 | controller |     - '          exit 1'
2026-03-14 09:56:26.714440 | controller |     - '        fi'
2026-03-14 09:56:26.714444 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.714448 | controller |     - '        '
2026-03-14 09:56:26.714453 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.714457 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.714461 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.714465 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.714470 | controller |       --timeout=300s'
2026-03-14 09:56:26.714474 | controller |     - '        '
2026-03-14 09:56:26.714478 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.714483 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.714487 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.714491 | controller |     - '        '
2026-03-14 09:56:26.714496 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.714500 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.714505 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.714509 | controller |     - '        '
2026-03-14 09:56:26.714513 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.714518 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.714522 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.714526 | controller |       available"'
2026-03-14 09:56:26.714531 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.714535 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.714539 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.714546 | controller |     - '          exit 1'
2026-03-14 09:56:26.714550 | controller |     - '        fi'
2026-03-14 09:56:26.714554 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.714559 | controller |     - '        '
2026-03-14 09:56:26.714563 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.714569 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.714574 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.714578 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.714582 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.714587 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.714591 | controller |       -- \'
2026-03-14 09:56:26.714609 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.714614 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.714618 | controller |     - '    logger.go:42: 09:42:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.714623 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.714627 | controller |     - '    logger.go:42: 09:42:37 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.714632 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.714636 | controller |       found'
2026-03-14 09:56:26.714647 | controller |     - '    logger.go:42: 09:42:38 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.714652 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.714656 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.714661 | controller |     - '        '
2026-03-14 09:56:26.714665 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.714669 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.714674 | controller |     - '        '
2026-03-14 09:56:26.714678 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.714682 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.714687 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.714691 | controller |     - '          exit 1'
2026-03-14 09:56:26.714695 | controller |     - '        fi'
2026-03-14 09:56:26.714700 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.714704 | controller |     - '        '
2026-03-14 09:56:26.714708 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.714713 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.714717 | controller |       | base64 -d)'
2026-03-14 09:56:26.714721 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.714726 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.714730 | controller |     - '          exit 1'
2026-03-14 09:56:26.714734 | controller |     - '        fi'
2026-03-14 09:56:26.714739 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.714743 | controller |     - '        '
2026-03-14 09:56:26.714747 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.714752 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.714756 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.714761 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.714770 | controller |       --timeout=300s'
2026-03-14 09:56:26.714776 | controller |     - '        '
2026-03-14 09:56:26.714782 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.714789 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.714794 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.714799 | controller |     - '        '
2026-03-14 09:56:26.714803 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.714808 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.714812 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.714816 | controller |     - '        '
2026-03-14 09:56:26.714821 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.714825 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.714829 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.714833 | controller |       available"'
2026-03-14 09:56:26.714838 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.714842 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.714846 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.714851 | controller |     - '          exit 1'
2026-03-14 09:56:26.714855 | controller |     - '        fi'
2026-03-14 09:56:26.714859 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.714864 | controller |     - '        '
2026-03-14 09:56:26.714868 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.714872 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.714877 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.714881 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.714885 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.714890 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.714894 | controller |       -- \'
2026-03-14 09:56:26.714898 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.714903 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.714907 | controller |     - '    logger.go:42: 09:42:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.714911 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.714916 | controller |     - '    logger.go:42: 09:42:38 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.714920 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.714924 | controller |       found'
2026-03-14 09:56:26.714929 | controller |     - '    logger.go:42: 09:42:39 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.714940 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.714945 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.714950 | controller |     - '        '
2026-03-14 09:56:26.714954 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.714959 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.714965 | controller |     - '        '
2026-03-14 09:56:26.714969 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.714974 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.714981 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.714985 | controller |     - '          exit 1'
2026-03-14 09:56:26.715003 | controller |     - '        fi'
2026-03-14 09:56:26.715010 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.715014 | controller |     - '        '
2026-03-14 09:56:26.715019 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.715023 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.715028 | controller |       | base64 -d)'
2026-03-14 09:56:26.715034 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.715039 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.715043 | controller |     - '          exit 1'
2026-03-14 09:56:26.715047 | controller |     - '        fi'
2026-03-14 09:56:26.715052 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.715056 | controller |     - '        '
2026-03-14 09:56:26.715060 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.715065 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.715069 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.715073 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.715078 | controller |       --timeout=300s'
2026-03-14 09:56:26.715082 | controller |     - '        '
2026-03-14 09:56:26.715087 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.715091 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.715095 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.715100 | controller |     - '        '
2026-03-14 09:56:26.715104 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.715108 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.715113 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.715117 | controller |     - '        '
2026-03-14 09:56:26.715122 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.715126 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.715130 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.715135 | controller |       available"'
2026-03-14 09:56:26.715139 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.715143 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.715148 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.715152 | controller |     - '          exit 1'
2026-03-14 09:56:26.715156 | controller |     - '        fi'
2026-03-14 09:56:26.715161 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.715165 | controller |     - '        '
2026-03-14 09:56:26.715169 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.715174 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.715178 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.715186 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.715192 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.715197 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.715201 | controller |       -- \'
2026-03-14 09:56:26.715206 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.715210 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.715214 | controller |     - '    logger.go:42: 09:42:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.715219 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.715223 | controller |     - '    logger.go:42: 09:42:39 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.715228 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.715232 | controller |       found'
2026-03-14 09:56:26.715237 | controller |     - '    logger.go:42: 09:42:40 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.715241 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.715253 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.715258 | controller |     - '        '
2026-03-14 09:56:26.715262 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.715267 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.715271 | controller |     - '        '
2026-03-14 09:56:26.715275 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.715280 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.715284 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.715289 | controller |     - '          exit 1'
2026-03-14 09:56:26.715293 | controller |     - '        fi'
2026-03-14 09:56:26.715297 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.715302 | controller |     - '        '
2026-03-14 09:56:26.715306 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.715313 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.715317 | controller |       | base64 -d)'
2026-03-14 09:56:26.715322 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.715326 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.715330 | controller |     - '          exit 1'
2026-03-14 09:56:26.715335 | controller |     - '        fi'
2026-03-14 09:56:26.715339 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.715344 | controller |     - '        '
2026-03-14 09:56:26.715348 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.715352 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.715357 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.715361 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.715365 | controller |       --timeout=300s'
2026-03-14 09:56:26.715370 | controller |     - '        '
2026-03-14 09:56:26.715374 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.715379 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.715383 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.715391 | controller |     - '        '
2026-03-14 09:56:26.715396 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.715400 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.715405 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.715409 | controller |     - '        '
2026-03-14 09:56:26.715413 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.715418 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.715422 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.715427 | controller |       available"'
2026-03-14 09:56:26.715431 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.715435 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.715440 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.715444 | controller |     - '          exit 1'
2026-03-14 09:56:26.715448 | controller |     - '        fi'
2026-03-14 09:56:26.715453 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.715457 | controller |     - '        '
2026-03-14 09:56:26.715461 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.715466 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.715470 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.715475 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.715479 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.715483 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.715490 | controller |       -- \'
2026-03-14 09:56:26.715495 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.715499 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.715504 | controller |     - '    logger.go:42: 09:42:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.715508 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.715512 | controller |     - '    logger.go:42: 09:42:40 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.715517 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.715521 | controller |       found'
2026-03-14 09:56:26.715525 | controller |     - '    logger.go:42: 09:42:41 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.715530 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.715534 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.715538 | controller |     - '        '
2026-03-14 09:56:26.715549 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.715553 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.715558 | controller |     - '        '
2026-03-14 09:56:26.715562 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.715567 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.715571 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.715575 | controller |     - '          exit 1'
2026-03-14 09:56:26.715580 | controller |     - '        fi'
2026-03-14 09:56:26.715584 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.715588 | controller |     - '        '
2026-03-14 09:56:26.715605 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.715614 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.715618 | controller |       | base64 -d)'
2026-03-14 09:56:26.715623 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.715627 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.715631 | controller |     - '          exit 1'
2026-03-14 09:56:26.715636 | controller |     - '        fi'
2026-03-14 09:56:26.715640 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.715645 | controller |     - '        '
2026-03-14 09:56:26.715649 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.715653 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.715658 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.715662 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.715667 | controller |       --timeout=300s'
2026-03-14 09:56:26.715671 | controller |     - '        '
2026-03-14 09:56:26.715675 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.715680 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.715684 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.715688 | controller |     - '        '
2026-03-14 09:56:26.715693 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.715697 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.715702 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.715706 | controller |     - '        '
2026-03-14 09:56:26.715710 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.715715 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.715719 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.715723 | controller |       available"'
2026-03-14 09:56:26.715728 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.715732 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.715736 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.715741 | controller |     - '          exit 1'
2026-03-14 09:56:26.715745 | controller |     - '        fi'
2026-03-14 09:56:26.715749 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.715755 | controller |     - '        '
2026-03-14 09:56:26.715761 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.715767 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.715773 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.715780 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.715786 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.715792 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.715798 | controller |       -- \'
2026-03-14 09:56:26.715803 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.715810 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.715817 | controller |     - '    logger.go:42: 09:42:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.715822 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.715826 | controller |     - '    logger.go:42: 09:42:41 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.715830 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.715835 | controller |       found'
2026-03-14 09:56:26.715839 | controller |     - '    logger.go:42: 09:42:42 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.715844 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.715848 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.715852 | controller |     - '        '
2026-03-14 09:56:26.715865 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.715870 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.715874 | controller |     - '        '
2026-03-14 09:56:26.715878 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.715883 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.715887 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.715892 | controller |     - '          exit 1'
2026-03-14 09:56:26.715896 | controller |     - '        fi'
2026-03-14 09:56:26.715901 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.715905 | controller |     - '        '
2026-03-14 09:56:26.715909 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.715914 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.715918 | controller |       | base64 -d)'
2026-03-14 09:56:26.715922 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.715927 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.715931 | controller |     - '          exit 1'
2026-03-14 09:56:26.715935 | controller |     - '        fi'
2026-03-14 09:56:26.715940 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.715944 | controller |     - '        '
2026-03-14 09:56:26.715948 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.715953 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.715957 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.715961 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.715966 | controller |       --timeout=300s'
2026-03-14 09:56:26.715970 | controller |     - '        '
2026-03-14 09:56:26.715975 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.715979 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.715983 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.715988 | controller |     - '        '
2026-03-14 09:56:26.715992 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.715996 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.716001 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.716005 | controller |     - '        '
2026-03-14 09:56:26.716010 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.716016 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.716020 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.716024 | controller |       available"'
2026-03-14 09:56:26.716029 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.716033 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.716037 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.716042 | controller |     - '          exit 1'
2026-03-14 09:56:26.716046 | controller |     - '        fi'
2026-03-14 09:56:26.716051 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.716055 | controller |     - '        '
2026-03-14 09:56:26.716059 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.716064 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.716068 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.716074 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.716079 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.716083 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.716087 | controller |       -- \'
2026-03-14 09:56:26.716092 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.716096 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.716101 | controller |     - '    logger.go:42: 09:42:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.716105 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.716109 | controller |     - '    logger.go:42: 09:42:43 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.716114 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.716118 | controller |       found'
2026-03-14 09:56:26.716122 | controller |     - '    logger.go:42: 09:42:44 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.716127 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.716131 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.716135 | controller |     - '        '
2026-03-14 09:56:26.716140 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.716150 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.716155 | controller |     - '        '
2026-03-14 09:56:26.716159 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.716164 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.716168 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.716173 | controller |     - '          exit 1'
2026-03-14 09:56:26.716177 | controller |     - '        fi'
2026-03-14 09:56:26.716181 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.716186 | controller |     - '        '
2026-03-14 09:56:26.716190 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.716194 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.716199 | controller |       | base64 -d)'
2026-03-14 09:56:26.716203 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.716207 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.716212 | controller |     - '          exit 1'
2026-03-14 09:56:26.716216 | controller |     - '        fi'
2026-03-14 09:56:26.716222 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.716226 | controller |     - '        '
2026-03-14 09:56:26.716231 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.716235 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.716239 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.716243 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.716248 | controller |       --timeout=300s'
2026-03-14 09:56:26.716252 | controller |     - '        '
2026-03-14 09:56:26.716256 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.716261 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.716265 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.716269 | controller |     - '        '
2026-03-14 09:56:26.716274 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.716278 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.716282 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.716286 | controller |     - '        '
2026-03-14 09:56:26.716291 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.716295 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.716299 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.716303 | controller |       available"'
2026-03-14 09:56:26.716308 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.716312 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.716316 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.716321 | controller |     - '          exit 1'
2026-03-14 09:56:26.716325 | controller |     - '        fi'
2026-03-14 09:56:26.716330 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.716334 | controller |     - '        '
2026-03-14 09:56:26.716338 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.716342 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.716347 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.716351 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.716355 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.716361 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.716366 | controller |       -- \'
2026-03-14 09:56:26.716370 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.716374 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.716379 | controller |     - '    logger.go:42: 09:42:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.716383 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.716387 | controller |     - '    logger.go:42: 09:42:44 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.716393 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.716399 | controller |       found'
2026-03-14 09:56:26.716403 | controller |     - '    logger.go:42: 09:42:45 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.716407 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.716412 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.716416 | controller |     - '        '
2026-03-14 09:56:26.716420 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.716425 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.716435 | controller |     - '        '
2026-03-14 09:56:26.716439 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.716443 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.716448 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.716452 | controller |     - '          exit 1'
2026-03-14 09:56:26.716456 | controller |     - '        fi'
2026-03-14 09:56:26.716461 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.716465 | controller |     - '        '
2026-03-14 09:56:26.716469 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.716473 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.716478 | controller |       | base64 -d)'
2026-03-14 09:56:26.716482 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.716486 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.716491 | controller |     - '          exit 1'
2026-03-14 09:56:26.716495 | controller |     - '        fi'
2026-03-14 09:56:26.716499 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.716503 | controller |     - '        '
2026-03-14 09:56:26.716508 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.716512 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.716516 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.716520 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.716525 | controller |       --timeout=300s'
2026-03-14 09:56:26.716529 | controller |     - '        '
2026-03-14 09:56:26.716533 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.716538 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.716542 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.716546 | controller |     - '        '
2026-03-14 09:56:26.716550 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.716555 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.716559 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.716563 | controller |     - '        '
2026-03-14 09:56:26.716568 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.716572 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.716576 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.716580 | controller |       available"'
2026-03-14 09:56:26.716585 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.716589 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.716611 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.716615 | controller |     - '          exit 1'
2026-03-14 09:56:26.716621 | controller |     - '        fi'
2026-03-14 09:56:26.716626 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.716630 | controller |     - '        '
2026-03-14 09:56:26.716634 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.716639 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.716643 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.716647 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.716651 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.716656 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.716660 | controller |       -- \'
2026-03-14 09:56:26.716664 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.716669 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.716673 | controller |     - '    logger.go:42: 09:42:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.716677 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.716682 | controller |     - '    logger.go:42: 09:42:45 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.716686 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.716690 | controller |       found'
2026-03-14 09:56:26.716694 | controller |     - '    logger.go:42: 09:42:46 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.716699 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.716703 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.716707 | controller |     - '        '
2026-03-14 09:56:26.716711 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.716716 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.716720 | controller |     - '        '
2026-03-14 09:56:26.716731 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.716737 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.716741 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.716746 | controller |     - '          exit 1'
2026-03-14 09:56:26.716750 | controller |     - '        fi'
2026-03-14 09:56:26.716755 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.716759 | controller |     - '        '
2026-03-14 09:56:26.716763 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.716768 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.716772 | controller |       | base64 -d)'
2026-03-14 09:56:26.716776 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.716780 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.716785 | controller |     - '          exit 1'
2026-03-14 09:56:26.716789 | controller |     - '        fi'
2026-03-14 09:56:26.716793 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.716798 | controller |     - '        '
2026-03-14 09:56:26.716802 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.716806 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.716810 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.716815 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.716819 | controller |       --timeout=300s'
2026-03-14 09:56:26.716825 | controller |     - '        '
2026-03-14 09:56:26.716830 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.716835 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.716840 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.716844 | controller |     - '        '
2026-03-14 09:56:26.716848 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.716853 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.716857 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.716861 | controller |     - '        '
2026-03-14 09:56:26.716866 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.716870 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.716874 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.716878 | controller |       available"'
2026-03-14 09:56:26.716883 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.716887 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.716891 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.716896 | controller |     - '          exit 1'
2026-03-14 09:56:26.716900 | controller |     - '        fi'
2026-03-14 09:56:26.716904 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.716908 | controller |     - '        '
2026-03-14 09:56:26.716913 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.716917 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.716921 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.716926 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.716930 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.716934 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.716938 | controller |       -- \'
2026-03-14 09:56:26.716943 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.716947 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.716951 | controller |     - '    logger.go:42: 09:42:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.716956 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.716960 | controller |     - '    logger.go:42: 09:42:46 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.716964 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.716970 | controller |       found'
2026-03-14 09:56:26.716974 | controller |     - '    logger.go:42: 09:42:47 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.716979 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.716983 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.716987 | controller |     - '        '
2026-03-14 09:56:26.716992 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.716996 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.717000 | controller |     - '        '
2026-03-14 09:56:26.717006 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.717017 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.717021 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.717026 | controller |     - '          exit 1'
2026-03-14 09:56:26.717030 | controller |     - '        fi'
2026-03-14 09:56:26.717034 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.717038 | controller |     - '        '
2026-03-14 09:56:26.717043 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.717047 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.717051 | controller |       | base64 -d)'
2026-03-14 09:56:26.717055 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.717060 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.717064 | controller |     - '          exit 1'
2026-03-14 09:56:26.717068 | controller |     - '        fi'
2026-03-14 09:56:26.717072 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.717077 | controller |     - '        '
2026-03-14 09:56:26.717081 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.717085 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.717090 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.717094 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.717098 | controller |       --timeout=300s'
2026-03-14 09:56:26.717102 | controller |     - '        '
2026-03-14 09:56:26.717107 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.717111 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.717115 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.717120 | controller |     - '        '
2026-03-14 09:56:26.717124 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.717128 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.717133 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.717139 | controller |     - '        '
2026-03-14 09:56:26.717143 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.717147 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.717152 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.717156 | controller |       available"'
2026-03-14 09:56:26.717160 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.717165 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.717169 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.717173 | controller |     - '          exit 1'
2026-03-14 09:56:26.717177 | controller |     - '        fi'
2026-03-14 09:56:26.717182 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.717186 | controller |     - '        '
2026-03-14 09:56:26.717190 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.717195 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.717199 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.717205 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.717209 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.717213 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.717218 | controller |       -- \'
2026-03-14 09:56:26.717222 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.717226 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.717230 | controller |     - '    logger.go:42: 09:42:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.717235 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.717239 | controller |     - '    logger.go:42: 09:42:47 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.717243 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.717248 | controller |       found'
2026-03-14 09:56:26.717252 | controller |     - '    logger.go:42: 09:42:48 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.717256 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.717260 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.717265 | controller |     - '        '
2026-03-14 09:56:26.717269 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.717273 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.717278 | controller |     - '        '
2026-03-14 09:56:26.717282 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.717286 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.717296 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.717301 | controller |     - '          exit 1'
2026-03-14 09:56:26.717305 | controller |     - '        fi'
2026-03-14 09:56:26.717309 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.717313 | controller |     - '        '
2026-03-14 09:56:26.717318 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.717322 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.717326 | controller |       | base64 -d)'
2026-03-14 09:56:26.717331 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.717335 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.717341 | controller |     - '          exit 1'
2026-03-14 09:56:26.717346 | controller |     - '        fi'
2026-03-14 09:56:26.717350 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.717355 | controller |     - '        '
2026-03-14 09:56:26.717361 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.717366 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.717370 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.717374 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.717378 | controller |       --timeout=300s'
2026-03-14 09:56:26.717382 | controller |     - '        '
2026-03-14 09:56:26.717387 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.717391 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.717395 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.717400 | controller |     - '        '
2026-03-14 09:56:26.717404 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.717410 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.717417 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.717421 | controller |     - '        '
2026-03-14 09:56:26.717425 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.717430 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.717434 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.717438 | controller |       available"'
2026-03-14 09:56:26.717442 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.717447 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.717451 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.717455 | controller |     - '          exit 1'
2026-03-14 09:56:26.717459 | controller |     - '        fi'
2026-03-14 09:56:26.717464 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.717468 | controller |     - '        '
2026-03-14 09:56:26.717472 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.717477 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.717481 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.717485 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.717490 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.717494 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.717498 | controller |       -- \'
2026-03-14 09:56:26.717502 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.717507 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.717511 | controller |     - '    logger.go:42: 09:42:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.717515 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.717520 | controller |     - '    logger.go:42: 09:42:48 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.717524 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.717528 | controller |       found'
2026-03-14 09:56:26.717533 | controller |     - '    logger.go:42: 09:42:49 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.717538 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.717543 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.717547 | controller |     - '        '
2026-03-14 09:56:26.717551 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.717556 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.717560 | controller |     - '        '
2026-03-14 09:56:26.717564 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.717568 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.717578 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.717583 | controller |     - '          exit 1'
2026-03-14 09:56:26.717587 | controller |     - '        fi'
2026-03-14 09:56:26.717591 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.717609 | controller |     - '        '
2026-03-14 09:56:26.717613 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.717620 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.717624 | controller |       | base64 -d)'
2026-03-14 09:56:26.717629 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.717633 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.717637 | controller |     - '          exit 1'
2026-03-14 09:56:26.717641 | controller |     - '        fi'
2026-03-14 09:56:26.717646 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.717650 | controller |     - '        '
2026-03-14 09:56:26.717654 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.717659 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.717663 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.717667 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.717671 | controller |       --timeout=300s'
2026-03-14 09:56:26.717676 | controller |     - '        '
2026-03-14 09:56:26.717680 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.717684 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.717689 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.717693 | controller |     - '        '
2026-03-14 09:56:26.717697 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.717701 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.717706 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.717710 | controller |     - '        '
2026-03-14 09:56:26.717714 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.717719 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.717723 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.717727 | controller |       available"'
2026-03-14 09:56:26.717732 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.717736 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.717740 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.717744 | controller |     - '          exit 1'
2026-03-14 09:56:26.717749 | controller |     - '        fi'
2026-03-14 09:56:26.717753 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.717757 | controller |     - '        '
2026-03-14 09:56:26.717762 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.717766 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.717770 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.717775 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.717779 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.717783 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.717787 | controller |       -- \'
2026-03-14 09:56:26.717794 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.717799 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.717804 | controller |     - '    logger.go:42: 09:42:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.717808 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.717812 | controller |     - '    logger.go:42: 09:42:50 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.717817 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.717821 | controller |       found'
2026-03-14 09:56:26.717826 | controller |     - '    logger.go:42: 09:42:51 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.717830 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.717834 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.717839 | controller |     - '        '
2026-03-14 09:56:26.717843 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.717847 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.717852 | controller |     - '        '
2026-03-14 09:56:26.717856 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.717860 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.717865 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.717869 | controller |     - '          exit 1'
2026-03-14 09:56:26.717873 | controller |     - '        fi'
2026-03-14 09:56:26.717884 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.717888 | controller |     - '        '
2026-03-14 09:56:26.717893 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.717899 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.717903 | controller |       | base64 -d)'
2026-03-14 09:56:26.717907 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.717912 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.717916 | controller |     - '          exit 1'
2026-03-14 09:56:26.717920 | controller |     - '        fi'
2026-03-14 09:56:26.717925 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.717929 | controller |     - '        '
2026-03-14 09:56:26.717933 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.717938 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.717942 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.717946 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.717950 | controller |       --timeout=300s'
2026-03-14 09:56:26.717955 | controller |     - '        '
2026-03-14 09:56:26.717959 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.717963 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.717968 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.717972 | controller |     - '        '
2026-03-14 09:56:26.717976 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.717980 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.717985 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.717989 | controller |     - '        '
2026-03-14 09:56:26.717993 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.717998 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.718005 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.718010 | controller |       available"'
2026-03-14 09:56:26.718014 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.718018 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.718023 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.718027 | controller |     - '          exit 1'
2026-03-14 09:56:26.718031 | controller |     - '        fi'
2026-03-14 09:56:26.718035 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.718040 | controller |     - '        '
2026-03-14 09:56:26.718044 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.718048 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.718053 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.718057 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.718061 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.718065 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.718070 | controller |       -- \'
2026-03-14 09:56:26.718076 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.718080 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.718084 | controller |     - '    logger.go:42: 09:42:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.718088 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.718093 | controller |     - '    logger.go:42: 09:42:51 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.718097 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.718101 | controller |       found'
2026-03-14 09:56:26.718106 | controller |     - '    logger.go:42: 09:42:52 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.718110 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.718114 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.718119 | controller |     - '        '
2026-03-14 09:56:26.718123 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.718129 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.718134 | controller |     - '        '
2026-03-14 09:56:26.718138 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.718142 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.718147 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.718151 | controller |     - '          exit 1'
2026-03-14 09:56:26.718155 | controller |     - '        fi'
2026-03-14 09:56:26.718159 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.718171 | controller |     - '        '
2026-03-14 09:56:26.718176 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.718180 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.718186 | controller |       | base64 -d)'
2026-03-14 09:56:26.718190 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.718195 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.718199 | controller |     - '          exit 1'
2026-03-14 09:56:26.718203 | controller |     - '        fi'
2026-03-14 09:56:26.718209 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.718215 | controller |     - '        '
2026-03-14 09:56:26.718220 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.718224 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.718228 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.718232 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.718237 | controller |       --timeout=300s'
2026-03-14 09:56:26.718241 | controller |     - '        '
2026-03-14 09:56:26.718245 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.718250 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.718254 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.718258 | controller |     - '        '
2026-03-14 09:56:26.718263 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.718267 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.718271 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.718275 | controller |     - '        '
2026-03-14 09:56:26.718280 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.718284 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.718288 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.718293 | controller |       available"'
2026-03-14 09:56:26.718297 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.718301 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.718305 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.718310 | controller |     - '          exit 1'
2026-03-14 09:56:26.718314 | controller |     - '        fi'
2026-03-14 09:56:26.718318 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.718322 | controller |     - '        '
2026-03-14 09:56:26.718327 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.718331 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.718335 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.718339 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.718344 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.718348 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.718352 | controller |       -- \'
2026-03-14 09:56:26.718357 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.718362 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.718367 | controller |     - '    logger.go:42: 09:42:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.718371 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.718375 | controller |     - '    logger.go:42: 09:42:52 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.718380 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.718384 | controller |       found'
2026-03-14 09:56:26.718388 | controller |     - '    logger.go:42: 09:42:53 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.718395 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.718399 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.718403 | controller |     - '        '
2026-03-14 09:56:26.718408 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.718412 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.718416 | controller |     - '        '
2026-03-14 09:56:26.718420 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.718425 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.718429 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.718433 | controller |     - '          exit 1'
2026-03-14 09:56:26.718438 | controller |     - '        fi'
2026-03-14 09:56:26.718442 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.718446 | controller |     - '        '
2026-03-14 09:56:26.718457 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.718461 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.718466 | controller |       | base64 -d)'
2026-03-14 09:56:26.718470 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.718474 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.718478 | controller |     - '          exit 1'
2026-03-14 09:56:26.718483 | controller |     - '        fi'
2026-03-14 09:56:26.718487 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.718491 | controller |     - '        '
2026-03-14 09:56:26.718496 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.718500 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.718504 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.718508 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.718513 | controller |       --timeout=300s'
2026-03-14 09:56:26.718517 | controller |     - '        '
2026-03-14 09:56:26.718521 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.718526 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.718530 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.718534 | controller |     - '        '
2026-03-14 09:56:26.718538 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.718543 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.718547 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.718551 | controller |     - '        '
2026-03-14 09:56:26.718556 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.718560 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.718564 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.718568 | controller |       available"'
2026-03-14 09:56:26.718573 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.718577 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.718581 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.718585 | controller |     - '          exit 1'
2026-03-14 09:56:26.718590 | controller |     - '        fi'
2026-03-14 09:56:26.718610 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.718616 | controller |     - '        '
2026-03-14 09:56:26.718621 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.718625 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.718629 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.718633 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.718638 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.718642 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.718646 | controller |       -- \'
2026-03-14 09:56:26.718651 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.718655 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.718659 | controller |     - '    logger.go:42: 09:42:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.718663 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.718668 | controller |     - '    logger.go:42: 09:42:53 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.718672 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.718676 | controller |       found'
2026-03-14 09:56:26.718681 | controller |     - '    logger.go:42: 09:42:54 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.718685 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.718692 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.718696 | controller |     - '        '
2026-03-14 09:56:26.718700 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.718705 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.718711 | controller |     - '        '
2026-03-14 09:56:26.718715 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.718720 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.718724 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.718728 | controller |     - '          exit 1'
2026-03-14 09:56:26.718733 | controller |     - '        fi'
2026-03-14 09:56:26.718737 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.718741 | controller |     - '        '
2026-03-14 09:56:26.718745 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.718756 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.718761 | controller |       | base64 -d)'
2026-03-14 09:56:26.718765 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.718769 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.718773 | controller |     - '          exit 1'
2026-03-14 09:56:26.718778 | controller |     - '        fi'
2026-03-14 09:56:26.718782 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.718786 | controller |     - '        '
2026-03-14 09:56:26.718791 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.718795 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.718799 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.718803 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.718808 | controller |       --timeout=300s'
2026-03-14 09:56:26.718812 | controller |     - '        '
2026-03-14 09:56:26.718818 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.718822 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.718827 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.718831 | controller |     - '        '
2026-03-14 09:56:26.718835 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.718840 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.718844 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.718848 | controller |     - '        '
2026-03-14 09:56:26.718853 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.718857 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.718861 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.718865 | controller |       available"'
2026-03-14 09:56:26.718870 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.718874 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.718878 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.718882 | controller |     - '          exit 1'
2026-03-14 09:56:26.718887 | controller |     - '        fi'
2026-03-14 09:56:26.718891 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.718895 | controller |     - '        '
2026-03-14 09:56:26.718900 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.718904 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.718908 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.718912 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.718917 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.718921 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.718925 | controller |       -- \'
2026-03-14 09:56:26.718930 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.718934 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.718938 | controller |     - '    logger.go:42: 09:42:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.718943 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.718947 | controller |     - '    logger.go:42: 09:42:54 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.718951 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.718955 | controller |       found'
2026-03-14 09:56:26.718960 | controller |     - '    logger.go:42: 09:42:55 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.718964 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.718968 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.718973 | controller |     - '        '
2026-03-14 09:56:26.718977 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.718981 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.718985 | controller |     - '        '
2026-03-14 09:56:26.718990 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.719013 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.719017 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.719022 | controller |     - '          exit 1'
2026-03-14 09:56:26.719026 | controller |     - '        fi'
2026-03-14 09:56:26.719030 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.719034 | controller |     - '        '
2026-03-14 09:56:26.719039 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.719050 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.719054 | controller |       | base64 -d)'
2026-03-14 09:56:26.719058 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.719063 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.719067 | controller |     - '          exit 1'
2026-03-14 09:56:26.719071 | controller |     - '        fi'
2026-03-14 09:56:26.719076 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.719080 | controller |     - '        '
2026-03-14 09:56:26.719084 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.719091 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.719095 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.719099 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.719104 | controller |       --timeout=300s'
2026-03-14 09:56:26.719108 | controller |     - '        '
2026-03-14 09:56:26.719112 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.719116 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.719121 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.719125 | controller |     - '        '
2026-03-14 09:56:26.719129 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.719134 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.719138 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.719142 | controller |     - '        '
2026-03-14 09:56:26.719146 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.719151 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.719155 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.719159 | controller |       available"'
2026-03-14 09:56:26.719164 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.719168 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.719182 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.719186 | controller |     - '          exit 1'
2026-03-14 09:56:26.719191 | controller |     - '        fi'
2026-03-14 09:56:26.719195 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.719199 | controller |     - '        '
2026-03-14 09:56:26.719203 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.719208 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.719212 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.719216 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.719224 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.719228 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.719232 | controller |       -- \'
2026-03-14 09:56:26.719236 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.719241 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.719245 | controller |     - '    logger.go:42: 09:42:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.719249 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.719253 | controller |     - '    logger.go:42: 09:42:55 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.719258 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.719262 | controller |       found'
2026-03-14 09:56:26.719266 | controller |     - '    logger.go:42: 09:42:56 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.719271 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.719275 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.719279 | controller |     - '        '
2026-03-14 09:56:26.719283 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.719288 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.719292 | controller |     - '        '
2026-03-14 09:56:26.719296 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.719302 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.719307 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.719311 | controller |     - '          exit 1'
2026-03-14 09:56:26.719317 | controller |     - '        fi'
2026-03-14 09:56:26.719321 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.719326 | controller |     - '        '
2026-03-14 09:56:26.719330 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.719334 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.719339 | controller |       | base64 -d)'
2026-03-14 09:56:26.719350 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.719356 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.719360 | controller |     - '          exit 1'
2026-03-14 09:56:26.719365 | controller |     - '        fi'
2026-03-14 09:56:26.719370 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.719376 | controller |     - '        '
2026-03-14 09:56:26.719380 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.719384 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.719389 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.719393 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.719397 | controller |       --timeout=300s'
2026-03-14 09:56:26.719402 | controller |     - '        '
2026-03-14 09:56:26.719406 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.719410 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.719415 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.719419 | controller |     - '        '
2026-03-14 09:56:26.719423 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.719429 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.719433 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.719440 | controller |     - '        '
2026-03-14 09:56:26.719444 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.719448 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.719452 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.719457 | controller |       available"'
2026-03-14 09:56:26.719461 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.719465 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.719470 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.719474 | controller |     - '          exit 1'
2026-03-14 09:56:26.719478 | controller |     - '        fi'
2026-03-14 09:56:26.719482 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.719487 | controller |     - '        '
2026-03-14 09:56:26.719491 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.719495 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.719499 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.719504 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.719508 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.719512 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.719518 | controller |       -- \'
2026-03-14 09:56:26.719523 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.719527 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.719531 | controller |     - '    logger.go:42: 09:42:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.719536 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.719540 | controller |     - '    logger.go:42: 09:42:57 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.719544 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.719548 | controller |       found'
2026-03-14 09:56:26.719553 | controller |     - '    logger.go:42: 09:42:58 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.719557 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.719561 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.719567 | controller |     - '        '
2026-03-14 09:56:26.719573 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.719578 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.719582 | controller |     - '        '
2026-03-14 09:56:26.719586 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.719591 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.719605 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.719612 | controller |     - '          exit 1'
2026-03-14 09:56:26.719616 | controller |     - '        fi'
2026-03-14 09:56:26.719620 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.719625 | controller |     - '        '
2026-03-14 09:56:26.719629 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.719633 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.719640 | controller |       | base64 -d)'
2026-03-14 09:56:26.719651 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.719656 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.719660 | controller |     - '          exit 1'
2026-03-14 09:56:26.719664 | controller |     - '        fi'
2026-03-14 09:56:26.719669 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.719673 | controller |     - '        '
2026-03-14 09:56:26.719677 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.719682 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.719686 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.719690 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.719694 | controller |       --timeout=300s'
2026-03-14 09:56:26.719699 | controller |     - '        '
2026-03-14 09:56:26.719703 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.719707 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.719712 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.719716 | controller |     - '        '
2026-03-14 09:56:26.719720 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.719724 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.719729 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.719733 | controller |     - '        '
2026-03-14 09:56:26.719737 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.719741 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.719746 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.719750 | controller |       available"'
2026-03-14 09:56:26.719754 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.719759 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.719763 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.719767 | controller |     - '          exit 1'
2026-03-14 09:56:26.719771 | controller |     - '        fi'
2026-03-14 09:56:26.719776 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.719780 | controller |     - '        '
2026-03-14 09:56:26.719784 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.719789 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.719793 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.719797 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.719801 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.719806 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.719810 | controller |       -- \'
2026-03-14 09:56:26.719814 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.719819 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.719825 | controller |     - '    logger.go:42: 09:42:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.719829 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.719834 | controller |     - '    logger.go:42: 09:42:58 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.719838 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.719842 | controller |       found'
2026-03-14 09:56:26.719846 | controller |     - '    logger.go:42: 09:42:59 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.719851 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.719855 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.719859 | controller |     - '        '
2026-03-14 09:56:26.719864 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.719868 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.719872 | controller |     - '        '
2026-03-14 09:56:26.719876 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.719881 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.719885 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.719889 | controller |     - '          exit 1'
2026-03-14 09:56:26.719894 | controller |     - '        fi'
2026-03-14 09:56:26.719898 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.719902 | controller |     - '        '
2026-03-14 09:56:26.719907 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.719911 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.719915 | controller |       | base64 -d)'
2026-03-14 09:56:26.719919 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.719932 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.719936 | controller |     - '          exit 1'
2026-03-14 09:56:26.719941 | controller |     - '        fi'
2026-03-14 09:56:26.719945 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.719949 | controller |     - '        '
2026-03-14 09:56:26.719953 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.719958 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.719962 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.719966 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.719971 | controller |       --timeout=300s'
2026-03-14 09:56:26.719975 | controller |     - '        '
2026-03-14 09:56:26.719979 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.719983 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.719988 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.719992 | controller |     - '        '
2026-03-14 09:56:26.719996 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.720001 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.720005 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.720009 | controller |     - '        '
2026-03-14 09:56:26.720013 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.720018 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.720022 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.720028 | controller |       available"'
2026-03-14 09:56:26.720033 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.720037 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.720041 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.720045 | controller |     - '          exit 1'
2026-03-14 09:56:26.720050 | controller |     - '        fi'
2026-03-14 09:56:26.720054 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.720058 | controller |     - '        '
2026-03-14 09:56:26.720063 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.720067 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.720071 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.720075 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.720080 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.720084 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.720088 | controller |       -- \'
2026-03-14 09:56:26.720093 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.720097 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.720101 | controller |     - '    logger.go:42: 09:42:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.720106 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.720110 | controller |     - '    logger.go:42: 09:42:59 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.720114 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.720118 | controller |       found'
2026-03-14 09:56:26.720123 | controller |     - '    logger.go:42: 09:43:00 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.720127 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.720131 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.720136 | controller |     - '        '
2026-03-14 09:56:26.720140 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.720144 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.720148 | controller |     - '        '
2026-03-14 09:56:26.720153 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.720157 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.720161 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.720166 | controller |     - '          exit 1'
2026-03-14 09:56:26.720170 | controller |     - '        fi'
2026-03-14 09:56:26.720174 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.720178 | controller |     - '        '
2026-03-14 09:56:26.720183 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.720194 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.720198 | controller |       | base64 -d)'
2026-03-14 09:56:26.720202 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.720207 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.720217 | controller |     - '          exit 1'
2026-03-14 09:56:26.720222 | controller |     - '        fi'
2026-03-14 09:56:26.720226 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.720230 | controller |     - '        '
2026-03-14 09:56:26.720236 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.720241 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.720245 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.720249 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.720254 | controller |       --timeout=300s'
2026-03-14 09:56:26.720258 | controller |     - '        '
2026-03-14 09:56:26.720262 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.720266 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.720271 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.720275 | controller |     - '        '
2026-03-14 09:56:26.720279 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.720284 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.720288 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.720292 | controller |     - '        '
2026-03-14 09:56:26.720296 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.720301 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.720305 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.720309 | controller |       available"'
2026-03-14 09:56:26.720314 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.720318 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.720322 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.720327 | controller |     - '          exit 1'
2026-03-14 09:56:26.720337 | controller |     - '        fi'
2026-03-14 09:56:26.720342 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.720346 | controller |     - '        '
2026-03-14 09:56:26.720350 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.720355 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.720359 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.720363 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.720368 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.720372 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.720377 | controller |       -- \'
2026-03-14 09:56:26.720381 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.720385 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.720390 | controller |     - '    logger.go:42: 09:43:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.720394 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.720399 | controller |     - '    logger.go:42: 09:43:00 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.720403 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.720408 | controller |       found'
2026-03-14 09:56:26.720412 | controller |     - '    logger.go:42: 09:43:01 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.720418 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.720422 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.720427 | controller |     - '        '
2026-03-14 09:56:26.720431 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.720435 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.720440 | controller |     - '        '
2026-03-14 09:56:26.720444 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.720449 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.720453 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.720457 | controller |     - '          exit 1'
2026-03-14 09:56:26.720462 | controller |     - '        fi'
2026-03-14 09:56:26.720466 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.720471 | controller |     - '        '
2026-03-14 09:56:26.720475 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.720481 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.720485 | controller |       | base64 -d)'
2026-03-14 09:56:26.720490 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.720494 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.720499 | controller |     - '          exit 1'
2026-03-14 09:56:26.720503 | controller |     - '        fi'
2026-03-14 09:56:26.720514 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.720519 | controller |     - '        '
2026-03-14 09:56:26.720523 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.720527 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.720532 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.720538 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.720542 | controller |       --timeout=300s'
2026-03-14 09:56:26.720547 | controller |     - '        '
2026-03-14 09:56:26.720551 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.720556 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.720563 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.720568 | controller |     - '        '
2026-03-14 09:56:26.720572 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.720576 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.720581 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.720585 | controller |     - '        '
2026-03-14 09:56:26.720589 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.720605 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.720612 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.720616 | controller |       available"'
2026-03-14 09:56:26.720620 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.720625 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.720629 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.720634 | controller |     - '          exit 1'
2026-03-14 09:56:26.720638 | controller |     - '        fi'
2026-03-14 09:56:26.720642 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.720649 | controller |     - '        '
2026-03-14 09:56:26.720653 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.720658 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.720662 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.720667 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.720671 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.720675 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.720680 | controller |       -- \'
2026-03-14 09:56:26.720684 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.720688 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.720693 | controller |     - '    logger.go:42: 09:43:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.720697 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.720702 | controller |     - '    logger.go:42: 09:43:01 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.720706 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.720710 | controller |       found'
2026-03-14 09:56:26.720715 | controller |     - '    logger.go:42: 09:43:02 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.720719 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.720723 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.720735 | controller |     - '        '
2026-03-14 09:56:26.720739 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.720744 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.720748 | controller |     - '        '
2026-03-14 09:56:26.720752 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.720757 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.720761 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.720765 | controller |     - '          exit 1'
2026-03-14 09:56:26.720770 | controller |     - '        fi'
2026-03-14 09:56:26.720774 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.720778 | controller |     - '        '
2026-03-14 09:56:26.720783 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.720787 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.720791 | controller |       | base64 -d)'
2026-03-14 09:56:26.720796 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.720800 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.720804 | controller |     - '          exit 1'
2026-03-14 09:56:26.720809 | controller |     - '        fi'
2026-03-14 09:56:26.720813 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.720825 | controller |     - '        '
2026-03-14 09:56:26.720829 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.720834 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.720838 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.720842 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.720847 | controller |       --timeout=300s'
2026-03-14 09:56:26.720851 | controller |     - '        '
2026-03-14 09:56:26.720856 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.720864 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.720869 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.720873 | controller |     - '        '
2026-03-14 09:56:26.720877 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.720882 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.720886 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.720891 | controller |     - '        '
2026-03-14 09:56:26.720895 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.720899 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.720904 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.720908 | controller |       available"'
2026-03-14 09:56:26.720912 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.720917 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.720921 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.720925 | controller |     - '          exit 1'
2026-03-14 09:56:26.720930 | controller |     - '        fi'
2026-03-14 09:56:26.720934 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.720938 | controller |     - '        '
2026-03-14 09:56:26.720943 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.720949 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.720955 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.720960 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.720966 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.720971 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.720977 | controller |       -- \'
2026-03-14 09:56:26.720982 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.720988 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.720993 | controller |     - '    logger.go:42: 09:43:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.720999 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.721005 | controller |     - '    logger.go:42: 09:43:02 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.721010 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.721016 | controller |       found'
2026-03-14 09:56:26.721022 | controller |     - '    logger.go:42: 09:43:03 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.721027 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.721033 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.721039 | controller |     - '        '
2026-03-14 09:56:26.721045 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.721050 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.721056 | controller |     - '        '
2026-03-14 09:56:26.721061 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.721069 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.721076 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.721082 | controller |     - '          exit 1'
2026-03-14 09:56:26.721089 | controller |     - '        fi'
2026-03-14 09:56:26.721094 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.721101 | controller |     - '        '
2026-03-14 09:56:26.721106 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.721115 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.721121 | controller |       | base64 -d)'
2026-03-14 09:56:26.721126 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.721132 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.721138 | controller |     - '          exit 1'
2026-03-14 09:56:26.721143 | controller |     - '        fi'
2026-03-14 09:56:26.721149 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.721154 | controller |     - '        '
2026-03-14 09:56:26.721169 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.721175 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.721181 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.721187 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.721206 | controller |       --timeout=300s'
2026-03-14 09:56:26.721212 | controller |     - '        '
2026-03-14 09:56:26.721218 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.721224 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.721235 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.721241 | controller |     - '        '
2026-03-14 09:56:26.721248 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.721253 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.721259 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.721264 | controller |     - '        '
2026-03-14 09:56:26.721270 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.721275 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.721281 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.721287 | controller |       available"'
2026-03-14 09:56:26.721292 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.721298 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.721304 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.721311 | controller |     - '          exit 1'
2026-03-14 09:56:26.721318 | controller |     - '        fi'
2026-03-14 09:56:26.721325 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.721333 | controller |     - '        '
2026-03-14 09:56:26.721339 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.721346 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.721352 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.721357 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.721362 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.721370 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.721376 | controller |       -- \'
2026-03-14 09:56:26.721382 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.721388 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.721394 | controller |     - '    logger.go:42: 09:43:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.721401 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.721406 | controller |     - '    logger.go:42: 09:43:03 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.721412 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.721418 | controller |       found'
2026-03-14 09:56:26.721424 | controller |     - '    logger.go:42: 09:43:04 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.721431 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.721436 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.721442 | controller |     - '        '
2026-03-14 09:56:26.721448 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.721454 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.721460 | controller |     - '        '
2026-03-14 09:56:26.721467 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.721473 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.721478 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.721485 | controller |     - '          exit 1'
2026-03-14 09:56:26.721491 | controller |     - '        fi'
2026-03-14 09:56:26.721497 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.721503 | controller |     - '        '
2026-03-14 09:56:26.721508 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.721515 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.721521 | controller |       | base64 -d)'
2026-03-14 09:56:26.721527 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.721533 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.721539 | controller |     - '          exit 1'
2026-03-14 09:56:26.721546 | controller |     - '        fi'
2026-03-14 09:56:26.721552 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.721558 | controller |     - '        '
2026-03-14 09:56:26.721564 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.721580 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.721588 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.721608 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.721618 | controller |       --timeout=300s'
2026-03-14 09:56:26.721628 | controller |     - '        '
2026-03-14 09:56:26.721634 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.721640 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.721646 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.721652 | controller |     - '        '
2026-03-14 09:56:26.721658 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.721665 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.721674 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.721680 | controller |     - '        '
2026-03-14 09:56:26.721686 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.721693 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.721699 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.721708 | controller |       available"'
2026-03-14 09:56:26.721714 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.721720 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.721740 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.721748 | controller |     - '          exit 1'
2026-03-14 09:56:26.721760 | controller |     - '        fi'
2026-03-14 09:56:26.721779 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.721785 | controller |     - '        '
2026-03-14 09:56:26.721792 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.721798 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.721806 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.721818 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.721825 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.721831 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.721839 | controller |       -- \'
2026-03-14 09:56:26.721850 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.721856 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.721863 | controller |     - '    logger.go:42: 09:43:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.721870 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.721880 | controller |     - '    logger.go:42: 09:43:05 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.721896 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.721902 | controller |       found'
2026-03-14 09:56:26.721908 | controller |     - '    logger.go:42: 09:43:06 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.721916 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.721928 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.721935 | controller |     - '        '
2026-03-14 09:56:26.721941 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.721949 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.721955 | controller |     - '        '
2026-03-14 09:56:26.721962 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.721968 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.721974 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.721980 | controller |     - '          exit 1'
2026-03-14 09:56:26.721986 | controller |     - '        fi'
2026-03-14 09:56:26.721992 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.721998 | controller |     - '        '
2026-03-14 09:56:26.722005 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.722010 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.722019 | controller |       | base64 -d)'
2026-03-14 09:56:26.722026 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.722032 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.722038 | controller |     - '          exit 1'
2026-03-14 09:56:26.722044 | controller |     - '        fi'
2026-03-14 09:56:26.722050 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.722057 | controller |     - '        '
2026-03-14 09:56:26.722063 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.722078 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.722085 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.722091 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.722098 | controller |       --timeout=300s'
2026-03-14 09:56:26.722104 | controller |     - '        '
2026-03-14 09:56:26.722110 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.722116 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.722122 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.722128 | controller |     - '        '
2026-03-14 09:56:26.722134 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.722141 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.722147 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.722153 | controller |     - '        '
2026-03-14 09:56:26.722159 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.722169 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.722175 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.722181 | controller |       available"'
2026-03-14 09:56:26.722187 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.722193 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.722201 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.722208 | controller |     - '          exit 1'
2026-03-14 09:56:26.722214 | controller |     - '        fi'
2026-03-14 09:56:26.722220 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.722225 | controller |     - '        '
2026-03-14 09:56:26.722232 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.722246 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.722252 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.722258 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.722264 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.722270 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.722276 | controller |       -- \'
2026-03-14 09:56:26.722283 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.722289 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.722295 | controller |     - '    logger.go:42: 09:43:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.722304 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.722311 | controller |     - '    logger.go:42: 09:43:06 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.722317 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.722323 | controller |       found'
2026-03-14 09:56:26.722329 | controller |     - '    logger.go:42: 09:43:07 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.722335 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.722342 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.722348 | controller |     - '        '
2026-03-14 09:56:26.722354 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.722360 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.722366 | controller |     - '        '
2026-03-14 09:56:26.722372 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.722378 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.722384 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.722390 | controller |     - '          exit 1'
2026-03-14 09:56:26.722396 | controller |     - '        fi'
2026-03-14 09:56:26.722402 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.722408 | controller |     - '        '
2026-03-14 09:56:26.722415 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.722420 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.722426 | controller |       | base64 -d)'
2026-03-14 09:56:26.722433 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.722439 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.722445 | controller |     - '          exit 1'
2026-03-14 09:56:26.722451 | controller |     - '        fi'
2026-03-14 09:56:26.722457 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.722464 | controller |     - '        '
2026-03-14 09:56:26.722470 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.722475 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.722497 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.722505 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.722511 | controller |       --timeout=300s'
2026-03-14 09:56:26.722518 | controller |     - '        '
2026-03-14 09:56:26.722524 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.722530 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.722536 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.722542 | controller |     - '        '
2026-03-14 09:56:26.722549 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.722555 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.722561 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.722567 | controller |     - '        '
2026-03-14 09:56:26.722573 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.722579 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.722585 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.722612 | controller |       available"'
2026-03-14 09:56:26.722623 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.722629 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.722635 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.722642 | controller |     - '          exit 1'
2026-03-14 09:56:26.722648 | controller |     - '        fi'
2026-03-14 09:56:26.722654 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.722660 | controller |     - '        '
2026-03-14 09:56:26.722666 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.722673 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.722679 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.722685 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.722691 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.722697 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.722703 | controller |       -- \'
2026-03-14 09:56:26.722709 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.722715 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.722720 | controller |     - '    logger.go:42: 09:43:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.722726 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.722732 | controller |     - '    logger.go:42: 09:43:07 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.722737 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.722743 | controller |       found'
2026-03-14 09:56:26.722749 | controller |     - '    logger.go:42: 09:43:08 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.722755 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.722761 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.722767 | controller |     - '        '
2026-03-14 09:56:26.722773 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.722779 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.722785 | controller |     - '        '
2026-03-14 09:56:26.722791 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.722796 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.722803 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.722809 | controller |     - '          exit 1'
2026-03-14 09:56:26.722814 | controller |     - '        fi'
2026-03-14 09:56:26.722820 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.722830 | controller |     - '        '
2026-03-14 09:56:26.722836 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.722841 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.722849 | controller |       | base64 -d)'
2026-03-14 09:56:26.722854 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.722859 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.722865 | controller |     - '          exit 1'
2026-03-14 09:56:26.722870 | controller |     - '        fi'
2026-03-14 09:56:26.722876 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.722882 | controller |     - '        '
2026-03-14 09:56:26.722887 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.722896 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.722901 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.722917 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.722924 | controller |       --timeout=300s'
2026-03-14 09:56:26.722931 | controller |     - '        '
2026-03-14 09:56:26.722937 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.722942 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.723039 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.723043 | controller |     - '        '
2026-03-14 09:56:26.723048 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.723052 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.723056 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.723061 | controller |     - '        '
2026-03-14 09:56:26.723065 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.723070 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.723074 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.723081 | controller |       available"'
2026-03-14 09:56:26.723086 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.723090 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.723094 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.723099 | controller |     - '          exit 1'
2026-03-14 09:56:26.723103 | controller |     - '        fi'
2026-03-14 09:56:26.723107 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.723112 | controller |     - '        '
2026-03-14 09:56:26.723116 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.723120 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.723124 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.723129 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.723133 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.723137 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.723142 | controller |       -- \'
2026-03-14 09:56:26.723146 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.723150 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.723155 | controller |     - '    logger.go:42: 09:43:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.723159 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.723164 | controller |     - '    logger.go:42: 09:43:08 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.723168 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.723172 | controller |       found'
2026-03-14 09:56:26.723177 | controller |     - '    logger.go:42: 09:43:09 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.723181 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.723187 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.723192 | controller |     - '        '
2026-03-14 09:56:26.723196 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.723201 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.723205 | controller |     - '        '
2026-03-14 09:56:26.723209 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.723213 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.723218 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.723222 | controller |     - '          exit 1'
2026-03-14 09:56:26.723226 | controller |     - '        fi'
2026-03-14 09:56:26.723231 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.723235 | controller |     - '        '
2026-03-14 09:56:26.723241 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.723245 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.723250 | controller |       | base64 -d)'
2026-03-14 09:56:26.723254 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.723258 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.723263 | controller |     - '          exit 1'
2026-03-14 09:56:26.723267 | controller |     - '        fi'
2026-03-14 09:56:26.723271 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.723275 | controller |     - '        '
2026-03-14 09:56:26.723280 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.723284 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.723288 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.723301 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.723306 | controller |       --timeout=300s'
2026-03-14 09:56:26.723310 | controller |     - '        '
2026-03-14 09:56:26.723315 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.723319 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.723323 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.723330 | controller |     - '        '
2026-03-14 09:56:26.723334 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.723338 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.723343 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.723347 | controller |     - '        '
2026-03-14 09:56:26.723351 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.723355 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.723360 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.723364 | controller |       available"'
2026-03-14 09:56:26.723368 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.723373 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.723377 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.723381 | controller |     - '          exit 1'
2026-03-14 09:56:26.723385 | controller |     - '        fi'
2026-03-14 09:56:26.723390 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.723394 | controller |     - '        '
2026-03-14 09:56:26.723400 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.723404 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.723408 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.723413 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.723417 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.723421 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.723426 | controller |       -- \'
2026-03-14 09:56:26.723430 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.723434 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.723438 | controller |     - '    logger.go:42: 09:43:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.723443 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.723447 | controller |     - '    logger.go:42: 09:43:09 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.723451 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.723456 | controller |       found'
2026-03-14 09:56:26.723460 | controller |     - '    logger.go:42: 09:43:10 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.723464 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.723469 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.723473 | controller |     - '        '
2026-03-14 09:56:26.723477 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.723482 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.723486 | controller |     - '        '
2026-03-14 09:56:26.723490 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.723494 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.723499 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.723503 | controller |     - '          exit 1'
2026-03-14 09:56:26.723507 | controller |     - '        fi'
2026-03-14 09:56:26.723512 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.723516 | controller |     - '        '
2026-03-14 09:56:26.723520 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.723525 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.723529 | controller |       | base64 -d)'
2026-03-14 09:56:26.723533 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.723537 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.723542 | controller |     - '          exit 1'
2026-03-14 09:56:26.723546 | controller |     - '        fi'
2026-03-14 09:56:26.723550 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.723554 | controller |     - '        '
2026-03-14 09:56:26.723559 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.723563 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.723567 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.723571 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.723576 | controller |       --timeout=300s'
2026-03-14 09:56:26.723586 | controller |     - '        '
2026-03-14 09:56:26.723591 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.723609 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.723627 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.723632 | controller |     - '        '
2026-03-14 09:56:26.723636 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.723641 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.723645 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.723649 | controller |     - '        '
2026-03-14 09:56:26.723654 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.723658 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.723662 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.723667 | controller |       available"'
2026-03-14 09:56:26.723671 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.723675 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.723679 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.723684 | controller |     - '          exit 1'
2026-03-14 09:56:26.723688 | controller |     - '        fi'
2026-03-14 09:56:26.723692 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.723699 | controller |     - '        '
2026-03-14 09:56:26.723703 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.723707 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.723711 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.723716 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.723720 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.723724 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.723729 | controller |       -- \'
2026-03-14 09:56:26.723733 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.723737 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.723741 | controller |     - '    logger.go:42: 09:43:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.723746 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.723750 | controller |     - '    logger.go:42: 09:43:10 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.723754 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.723759 | controller |       found'
2026-03-14 09:56:26.723763 | controller |     - '    logger.go:42: 09:43:11 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.723767 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.723771 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.723776 | controller |     - '        '
2026-03-14 09:56:26.723780 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.723784 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.723789 | controller |     - '        '
2026-03-14 09:56:26.723793 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.723797 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.723803 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.723808 | controller |     - '          exit 1'
2026-03-14 09:56:26.723812 | controller |     - '        fi'
2026-03-14 09:56:26.723816 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.723820 | controller |     - '        '
2026-03-14 09:56:26.723825 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.723831 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.723835 | controller |       | base64 -d)'
2026-03-14 09:56:26.723839 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.723844 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.723848 | controller |     - '          exit 1'
2026-03-14 09:56:26.723852 | controller |     - '        fi'
2026-03-14 09:56:26.723857 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.723861 | controller |     - '        '
2026-03-14 09:56:26.723865 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.723869 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.723874 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.723878 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.723882 | controller |       --timeout=300s'
2026-03-14 09:56:26.723887 | controller |     - '        '
2026-03-14 09:56:26.723899 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.723903 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.723908 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.723912 | controller |     - '        '
2026-03-14 09:56:26.723916 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.723920 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.723925 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.723929 | controller |     - '        '
2026-03-14 09:56:26.723933 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.723937 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.723942 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.723946 | controller |       available"'
2026-03-14 09:56:26.723950 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.723954 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.723959 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.723963 | controller |     - '          exit 1'
2026-03-14 09:56:26.723967 | controller |     - '        fi'
2026-03-14 09:56:26.723971 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.723975 | controller |     - '        '
2026-03-14 09:56:26.723980 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.723984 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.723988 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.723992 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.723997 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.724003 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.724008 | controller |       -- \'
2026-03-14 09:56:26.724012 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.724016 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.724021 | controller |     - '    logger.go:42: 09:43:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.724025 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.724029 | controller |     - '    logger.go:42: 09:43:11 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.724034 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.724038 | controller |       found'
2026-03-14 09:56:26.724042 | controller |     - '    logger.go:42: 09:43:12 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.724046 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.724051 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.724055 | controller |     - '        '
2026-03-14 09:56:26.724059 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.724063 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.724068 | controller |     - '        '
2026-03-14 09:56:26.724072 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.724076 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.724081 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.724085 | controller |     - '          exit 1'
2026-03-14 09:56:26.724089 | controller |     - '        fi'
2026-03-14 09:56:26.724093 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.724098 | controller |     - '        '
2026-03-14 09:56:26.724102 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.724106 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.724111 | controller |       | base64 -d)'
2026-03-14 09:56:26.724115 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.724119 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.724123 | controller |     - '          exit 1'
2026-03-14 09:56:26.724128 | controller |     - '        fi'
2026-03-14 09:56:26.724132 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.724136 | controller |     - '        '
2026-03-14 09:56:26.724140 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.724145 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.724149 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.724153 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.724157 | controller |       --timeout=300s'
2026-03-14 09:56:26.724162 | controller |     - '        '
2026-03-14 09:56:26.724172 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.724176 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.724180 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.724185 | controller |     - '        '
2026-03-14 09:56:26.724189 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.724194 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.724201 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.724205 | controller |     - '        '
2026-03-14 09:56:26.724209 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.724214 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.724218 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.724222 | controller |       available"'
2026-03-14 09:56:26.724226 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.724231 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.724235 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.724239 | controller |     - '          exit 1'
2026-03-14 09:56:26.724244 | controller |     - '        fi'
2026-03-14 09:56:26.724248 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.724252 | controller |     - '        '
2026-03-14 09:56:26.724259 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.724265 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.724269 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.724273 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.724277 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.724283 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.724289 | controller |       -- \'
2026-03-14 09:56:26.724293 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.724297 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.724302 | controller |     - '    logger.go:42: 09:43:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.724306 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.724310 | controller |     - '    logger.go:42: 09:43:13 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.724315 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.724319 | controller |       found'
2026-03-14 09:56:26.724323 | controller |     - '    logger.go:42: 09:43:14 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.724327 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.724332 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.724336 | controller |     - '        '
2026-03-14 09:56:26.724340 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.724345 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.724349 | controller |     - '        '
2026-03-14 09:56:26.724353 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.724357 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.724362 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.724366 | controller |     - '          exit 1'
2026-03-14 09:56:26.724370 | controller |     - '        fi'
2026-03-14 09:56:26.724374 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.724379 | controller |     - '        '
2026-03-14 09:56:26.724383 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.724387 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.724391 | controller |       | base64 -d)'
2026-03-14 09:56:26.724397 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.724401 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.724406 | controller |     - '          exit 1'
2026-03-14 09:56:26.724410 | controller |     - '        fi'
2026-03-14 09:56:26.724414 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.724418 | controller |     - '        '
2026-03-14 09:56:26.724423 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.724427 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.724431 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.724435 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.724440 | controller |       --timeout=300s'
2026-03-14 09:56:26.724444 | controller |     - '        '
2026-03-14 09:56:26.724448 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.724459 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.724463 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.724468 | controller |     - '        '
2026-03-14 09:56:26.724472 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.724476 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.724481 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.724485 | controller |     - '        '
2026-03-14 09:56:26.724489 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.724494 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.724498 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.724502 | controller |       available"'
2026-03-14 09:56:26.724506 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.724511 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.724515 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.724519 | controller |     - '          exit 1'
2026-03-14 09:56:26.724523 | controller |     - '        fi'
2026-03-14 09:56:26.724528 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.724532 | controller |     - '        '
2026-03-14 09:56:26.724536 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.724541 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.724545 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.724549 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.724553 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.724558 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.724562 | controller |       -- \'
2026-03-14 09:56:26.724566 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.724572 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.724576 | controller |     - '    logger.go:42: 09:43:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.724582 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.724587 | controller |     - '    logger.go:42: 09:43:14 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.724591 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.724609 | controller |       found'
2026-03-14 09:56:26.724614 | controller |     - '    logger.go:42: 09:43:15 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.724618 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.724622 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.724626 | controller |     - '        '
2026-03-14 09:56:26.724631 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.724635 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.724639 | controller |     - '        '
2026-03-14 09:56:26.724644 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.724648 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.724652 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.724656 | controller |     - '          exit 1'
2026-03-14 09:56:26.724661 | controller |     - '        fi'
2026-03-14 09:56:26.724665 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.724669 | controller |     - '        '
2026-03-14 09:56:26.724673 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.724678 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.724682 | controller |       | base64 -d)'
2026-03-14 09:56:26.724686 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.724690 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.724695 | controller |     - '          exit 1'
2026-03-14 09:56:26.724699 | controller |     - '        fi'
2026-03-14 09:56:26.724703 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.724707 | controller |     - '        '
2026-03-14 09:56:26.724712 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.724716 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.724720 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.724725 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.724729 | controller |       --timeout=300s'
2026-03-14 09:56:26.724733 | controller |     - '        '
2026-03-14 09:56:26.724737 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.724748 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.724753 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.724758 | controller |     - '        '
2026-03-14 09:56:26.724762 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.724766 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.724770 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.724775 | controller |     - '        '
2026-03-14 09:56:26.724779 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.724783 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.724787 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.724792 | controller |       available"'
2026-03-14 09:56:26.724798 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.724802 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.724807 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.724811 | controller |     - '          exit 1'
2026-03-14 09:56:26.724815 | controller |     - '        fi'
2026-03-14 09:56:26.724819 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.724824 | controller |     - '        '
2026-03-14 09:56:26.724828 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.724832 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.724836 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.724841 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.724847 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.724851 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.724855 | controller |       -- \'
2026-03-14 09:56:26.724860 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.724864 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.724868 | controller |     - '    logger.go:42: 09:43:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.724872 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.724877 | controller |     - '    logger.go:42: 09:43:15 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.724881 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.724885 | controller |       found'
2026-03-14 09:56:26.724889 | controller |     - '    logger.go:42: 09:43:16 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.724894 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.724898 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.724902 | controller |     - '        '
2026-03-14 09:56:26.724906 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.724911 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.724915 | controller |     - '        '
2026-03-14 09:56:26.724919 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.724923 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.724928 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.724932 | controller |     - '          exit 1'
2026-03-14 09:56:26.724936 | controller |     - '        fi'
2026-03-14 09:56:26.724941 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.724945 | controller |     - '        '
2026-03-14 09:56:26.724949 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.724953 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.724958 | controller |       | base64 -d)'
2026-03-14 09:56:26.724962 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.724966 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.724970 | controller |     - '          exit 1'
2026-03-14 09:56:26.724975 | controller |     - '        fi'
2026-03-14 09:56:26.724979 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.724983 | controller |     - '        '
2026-03-14 09:56:26.724987 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.724992 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.724998 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.725002 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.725007 | controller |       --timeout=300s'
2026-03-14 09:56:26.725011 | controller |     - '        '
2026-03-14 09:56:26.725015 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.725019 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.725030 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.725034 | controller |     - '        '
2026-03-14 09:56:26.725038 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.725043 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.725047 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.725051 | controller |     - '        '
2026-03-14 09:56:26.725056 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.725060 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.725064 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.725069 | controller |       available"'
2026-03-14 09:56:26.725073 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.725077 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.725083 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.725088 | controller |     - '          exit 1'
2026-03-14 09:56:26.725092 | controller |     - '        fi'
2026-03-14 09:56:26.725096 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.725100 | controller |     - '        '
2026-03-14 09:56:26.725105 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.725109 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.725113 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.725117 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.725122 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.725126 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.725130 | controller |       -- \'
2026-03-14 09:56:26.725135 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.725139 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.725143 | controller |     - '    logger.go:42: 09:43:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.725147 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.725152 | controller |     - '    logger.go:42: 09:43:16 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.725156 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.725160 | controller |       found'
2026-03-14 09:56:26.725164 | controller |     - '    logger.go:42: 09:43:17 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.725169 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.725173 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.725179 | controller |     - '        '
2026-03-14 09:56:26.725183 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.725187 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.725192 | controller |     - '        '
2026-03-14 09:56:26.725196 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.725200 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.725205 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.725209 | controller |     - '          exit 1'
2026-03-14 09:56:26.725213 | controller |     - '        fi'
2026-03-14 09:56:26.725218 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.725222 | controller |     - '        '
2026-03-14 09:56:26.725226 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.725230 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.725235 | controller |       | base64 -d)'
2026-03-14 09:56:26.725239 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.725243 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.725248 | controller |     - '          exit 1'
2026-03-14 09:56:26.725252 | controller |     - '        fi'
2026-03-14 09:56:26.725256 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.725260 | controller |     - '        '
2026-03-14 09:56:26.725265 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.725269 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.725273 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.725278 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.725282 | controller |       --timeout=300s'
2026-03-14 09:56:26.725286 | controller |     - '        '
2026-03-14 09:56:26.725291 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.725295 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.725307 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.725312 | controller |     - '        '
2026-03-14 09:56:26.725316 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.725320 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.725326 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.725331 | controller |     - '        '
2026-03-14 09:56:26.725335 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.725339 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.725343 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.725348 | controller |       available"'
2026-03-14 09:56:26.725352 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.725356 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.725360 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.725365 | controller |     - '          exit 1'
2026-03-14 09:56:26.725369 | controller |     - '        fi'
2026-03-14 09:56:26.725373 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.725378 | controller |     - '        '
2026-03-14 09:56:26.725382 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.725389 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.725394 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.725398 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.725402 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.725406 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.725411 | controller |       -- \'
2026-03-14 09:56:26.725415 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.725419 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.725425 | controller |     - '    logger.go:42: 09:43:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.725429 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.725434 | controller |     - '    logger.go:42: 09:43:17 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.725438 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.725442 | controller |       found'
2026-03-14 09:56:26.725446 | controller |     - '    logger.go:42: 09:43:18 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.725451 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.725455 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.725459 | controller |     - '        '
2026-03-14 09:56:26.725464 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.725468 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.725472 | controller |     - '        '
2026-03-14 09:56:26.725476 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.725481 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.725485 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.725489 | controller |     - '          exit 1'
2026-03-14 09:56:26.725493 | controller |     - '        fi'
2026-03-14 09:56:26.725498 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.725502 | controller |     - '        '
2026-03-14 09:56:26.725506 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.725511 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.725515 | controller |       | base64 -d)'
2026-03-14 09:56:26.725519 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.725523 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.725528 | controller |     - '          exit 1'
2026-03-14 09:56:26.725532 | controller |     - '        fi'
2026-03-14 09:56:26.725536 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.725540 | controller |     - '        '
2026-03-14 09:56:26.725545 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.725549 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.725553 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.725558 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.725562 | controller |       --timeout=300s'
2026-03-14 09:56:26.725566 | controller |     - '        '
2026-03-14 09:56:26.725571 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.725585 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.725590 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.725624 | controller |     - '        '
2026-03-14 09:56:26.725632 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.725636 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.725640 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.725645 | controller |     - '        '
2026-03-14 09:56:26.725649 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.725653 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.725657 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.725662 | controller |       available"'
2026-03-14 09:56:26.725666 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.725670 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.725674 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.725679 | controller |     - '          exit 1'
2026-03-14 09:56:26.725683 | controller |     - '        fi'
2026-03-14 09:56:26.725687 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.725692 | controller |     - '        '
2026-03-14 09:56:26.725696 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.725700 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.725704 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.725709 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.725713 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.725717 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.725721 | controller |       -- \'
2026-03-14 09:56:26.725726 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.725730 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.725734 | controller |     - '    logger.go:42: 09:43:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.725738 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.725743 | controller |     - '    logger.go:42: 09:43:18 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.725747 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.725751 | controller |       found'
2026-03-14 09:56:26.725756 | controller |     - '    logger.go:42: 09:43:19 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.725760 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.725764 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.725768 | controller |     - '        '
2026-03-14 09:56:26.725773 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.725777 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.725781 | controller |     - '        '
2026-03-14 09:56:26.725786 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.725790 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.725794 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.725800 | controller |     - '          exit 1'
2026-03-14 09:56:26.725805 | controller |     - '        fi'
2026-03-14 09:56:26.725809 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.725813 | controller |     - '        '
2026-03-14 09:56:26.725818 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.725822 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.725826 | controller |       | base64 -d)'
2026-03-14 09:56:26.725830 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.725835 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.725839 | controller |     - '          exit 1'
2026-03-14 09:56:26.725843 | controller |     - '        fi'
2026-03-14 09:56:26.725848 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.725852 | controller |     - '        '
2026-03-14 09:56:26.725856 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.725860 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.725865 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.725869 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.725873 | controller |       --timeout=300s'
2026-03-14 09:56:26.725877 | controller |     - '        '
2026-03-14 09:56:26.725882 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.725886 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.725890 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.725895 | controller |     - '        '
2026-03-14 09:56:26.725906 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.725910 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.725915 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.725919 | controller |     - '        '
2026-03-14 09:56:26.725923 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.725928 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.725932 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.725936 | controller |       available"'
2026-03-14 09:56:26.725941 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.725945 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.725949 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.725953 | controller |     - '          exit 1'
2026-03-14 09:56:26.725958 | controller |     - '        fi'
2026-03-14 09:56:26.725962 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.725966 | controller |     - '        '
2026-03-14 09:56:26.725971 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.725975 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.725979 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.725983 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.725988 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.725994 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.725998 | controller |       -- \'
2026-03-14 09:56:26.726002 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.726007 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.726011 | controller |     - '    logger.go:42: 09:43:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.726015 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.726019 | controller |     - '    logger.go:42: 09:43:20 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.726026 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.726030 | controller |       found'
2026-03-14 09:56:26.726034 | controller |     - '    logger.go:42: 09:43:21 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.726038 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.726043 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.726047 | controller |     - '        '
2026-03-14 09:56:26.726051 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.726056 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.726060 | controller |     - '        '
2026-03-14 09:56:26.726064 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.726068 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.726073 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.726077 | controller |     - '          exit 1'
2026-03-14 09:56:26.726081 | controller |     - '        fi'
2026-03-14 09:56:26.726085 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.726090 | controller |     - '        '
2026-03-14 09:56:26.726094 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.726098 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.726102 | controller |       | base64 -d)'
2026-03-14 09:56:26.726107 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.726111 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.726115 | controller |     - '          exit 1'
2026-03-14 09:56:26.726119 | controller |     - '        fi'
2026-03-14 09:56:26.726124 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.726128 | controller |     - '        '
2026-03-14 09:56:26.726132 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.726136 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.726141 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.726145 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.726149 | controller |       --timeout=300s'
2026-03-14 09:56:26.726153 | controller |     - '        '
2026-03-14 09:56:26.726158 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.726162 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.726166 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.726170 | controller |     - '        '
2026-03-14 09:56:26.726181 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.726185 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.726190 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.726196 | controller |     - '        '
2026-03-14 09:56:26.726200 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.726204 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.726209 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.726213 | controller |       available"'
2026-03-14 09:56:26.726217 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.726222 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.726226 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.726230 | controller |     - '          exit 1'
2026-03-14 09:56:26.726234 | controller |     - '        fi'
2026-03-14 09:56:26.726239 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.726243 | controller |     - '        '
2026-03-14 09:56:26.726247 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.726251 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.726256 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.726260 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.726264 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.726268 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.726273 | controller |       -- \'
2026-03-14 09:56:26.726277 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.726283 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.726287 | controller |     - '    logger.go:42: 09:43:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.726292 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.726296 | controller |     - '    logger.go:42: 09:43:21 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.726300 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.726304 | controller |       found'
2026-03-14 09:56:26.726309 | controller |     - '    logger.go:42: 09:43:22 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.726313 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.726317 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.726322 | controller |     - '        '
2026-03-14 09:56:26.726326 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.726330 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.726335 | controller |     - '        '
2026-03-14 09:56:26.726339 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.726343 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.726347 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.726352 | controller |     - '          exit 1'
2026-03-14 09:56:26.726356 | controller |     - '        fi'
2026-03-14 09:56:26.726360 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.726365 | controller |     - '        '
2026-03-14 09:56:26.726369 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.726375 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.726379 | controller |       | base64 -d)'
2026-03-14 09:56:26.726384 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.726390 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.726394 | controller |     - '          exit 1'
2026-03-14 09:56:26.726400 | controller |     - '        fi'
2026-03-14 09:56:26.726404 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.726408 | controller |     - '        '
2026-03-14 09:56:26.726413 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.726417 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.726421 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.726425 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.726430 | controller |       --timeout=300s'
2026-03-14 09:56:26.726434 | controller |     - '        '
2026-03-14 09:56:26.726438 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.726443 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.726447 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.726451 | controller |     - '        '
2026-03-14 09:56:26.726456 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.726466 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.726470 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.726475 | controller |     - '        '
2026-03-14 09:56:26.726479 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.726483 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.726488 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.726492 | controller |       available"'
2026-03-14 09:56:26.726496 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.726501 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.726505 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.726509 | controller |     - '          exit 1'
2026-03-14 09:56:26.726513 | controller |     - '        fi'
2026-03-14 09:56:26.726518 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.726522 | controller |     - '        '
2026-03-14 09:56:26.726526 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.726531 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.726535 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.726539 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.726543 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.726548 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.726552 | controller |       -- \'
2026-03-14 09:56:26.726556 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.726560 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.726565 | controller |     - '    logger.go:42: 09:43:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.726569 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.726575 | controller |     - '    logger.go:42: 09:43:22 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.726579 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.726586 | controller |       found'
2026-03-14 09:56:26.726590 | controller |     - '    logger.go:42: 09:43:23 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.726605 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.726613 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.726620 | controller |     - '        '
2026-03-14 09:56:26.726626 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.726632 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.726637 | controller |     - '        '
2026-03-14 09:56:26.726643 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.726648 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.726654 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.726660 | controller |     - '          exit 1'
2026-03-14 09:56:26.726666 | controller |     - '        fi'
2026-03-14 09:56:26.726671 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.726677 | controller |     - '        '
2026-03-14 09:56:26.726683 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.726688 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.726694 | controller |       | base64 -d)'
2026-03-14 09:56:26.726700 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.726705 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.726711 | controller |     - '          exit 1'
2026-03-14 09:56:26.726717 | controller |     - '        fi'
2026-03-14 09:56:26.726723 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.726728 | controller |     - '        '
2026-03-14 09:56:26.726733 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.726737 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.726743 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.726747 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.726751 | controller |       --timeout=300s'
2026-03-14 09:56:26.726756 | controller |     - '        '
2026-03-14 09:56:26.726760 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.726764 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.726769 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.726773 | controller |     - '        '
2026-03-14 09:56:26.726778 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.726791 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.726796 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.726800 | controller |     - '        '
2026-03-14 09:56:26.726805 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.726809 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.726813 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.726818 | controller |       available"'
2026-03-14 09:56:26.726822 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.726831 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.726835 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.726839 | controller |     - '          exit 1'
2026-03-14 09:56:26.726845 | controller |     - '        fi'
2026-03-14 09:56:26.726851 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.726857 | controller |     - '        '
2026-03-14 09:56:26.726863 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.726869 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.726873 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.726878 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.726882 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.726887 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.726891 | controller |       -- \'
2026-03-14 09:56:26.726895 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.726900 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.726904 | controller |     - '    logger.go:42: 09:43:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.726908 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.726913 | controller |     - '    logger.go:42: 09:43:23 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.726917 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.726922 | controller |       found'
2026-03-14 09:56:26.726926 | controller |     - '    logger.go:42: 09:43:24 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.726930 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.726935 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.726939 | controller |     - '        '
2026-03-14 09:56:26.726944 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.726948 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.726952 | controller |     - '        '
2026-03-14 09:56:26.726957 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.726961 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.726965 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.726972 | controller |     - '          exit 1'
2026-03-14 09:56:26.726977 | controller |     - '        fi'
2026-03-14 09:56:26.726981 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.726987 | controller |     - '        '
2026-03-14 09:56:26.726992 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.727011 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.727018 | controller |       | base64 -d)'
2026-03-14 09:56:26.727023 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.727027 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.727031 | controller |     - '          exit 1'
2026-03-14 09:56:26.727036 | controller |     - '        fi'
2026-03-14 09:56:26.727040 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.727045 | controller |     - '        '
2026-03-14 09:56:26.727049 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.727053 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.727060 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.727064 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.727068 | controller |       --timeout=300s'
2026-03-14 09:56:26.727073 | controller |     - '        '
2026-03-14 09:56:26.727077 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.727081 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.727085 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.727090 | controller |     - '        '
2026-03-14 09:56:26.727094 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.727098 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.727111 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.727115 | controller |     - '        '
2026-03-14 09:56:26.727119 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.727124 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.727128 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.727132 | controller |       available"'
2026-03-14 09:56:26.727137 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.727141 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.727145 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.727149 | controller |     - '          exit 1'
2026-03-14 09:56:26.727154 | controller |     - '        fi'
2026-03-14 09:56:26.727158 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.727162 | controller |     - '        '
2026-03-14 09:56:26.727167 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.727171 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.727175 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.727180 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.727184 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.727188 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.727192 | controller |       -- \'
2026-03-14 09:56:26.727199 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.727203 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.727208 | controller |     - '    logger.go:42: 09:43:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.727212 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.727216 | controller |     - '    logger.go:42: 09:43:24 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.727221 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.727225 | controller |       found'
2026-03-14 09:56:26.727229 | controller |     - '    logger.go:42: 09:43:25 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.727235 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.727240 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.727244 | controller |     - '        '
2026-03-14 09:56:26.727249 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.727255 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.727259 | controller |     - '        '
2026-03-14 09:56:26.727263 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.727268 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.727272 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.727276 | controller |     - '          exit 1'
2026-03-14 09:56:26.727281 | controller |     - '        fi'
2026-03-14 09:56:26.727285 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.727289 | controller |     - '        '
2026-03-14 09:56:26.727294 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.727298 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.727302 | controller |       | base64 -d)'
2026-03-14 09:56:26.727306 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.727311 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.727315 | controller |     - '          exit 1'
2026-03-14 09:56:26.727319 | controller |     - '        fi'
2026-03-14 09:56:26.727324 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.727328 | controller |     - '        '
2026-03-14 09:56:26.727332 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.727337 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.727341 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.727345 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.727349 | controller |       --timeout=300s'
2026-03-14 09:56:26.727354 | controller |     - '        '
2026-03-14 09:56:26.727358 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.727362 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.727367 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.727371 | controller |     - '        '
2026-03-14 09:56:26.727375 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.727380 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.727390 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.727396 | controller |     - '        '
2026-03-14 09:56:26.727401 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.727405 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.727409 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.727414 | controller |       available"'
2026-03-14 09:56:26.727418 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.727422 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.727427 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.727431 | controller |     - '          exit 1'
2026-03-14 09:56:26.727435 | controller |     - '        fi'
2026-03-14 09:56:26.727440 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.727444 | controller |     - '        '
2026-03-14 09:56:26.727448 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.727456 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.727460 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.727464 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.727469 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.727473 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.727478 | controller |       -- \'
2026-03-14 09:56:26.727482 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.727486 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.727491 | controller |     - '    logger.go:42: 09:43:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.727495 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.727499 | controller |     - '    logger.go:42: 09:43:25 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.727504 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.727508 | controller |       found'
2026-03-14 09:56:26.727512 | controller |     - '    logger.go:42: 09:43:26 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.727517 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.727521 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.727525 | controller |     - '        '
2026-03-14 09:56:26.727530 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.727534 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.727538 | controller |     - '        '
2026-03-14 09:56:26.727545 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.727549 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.727553 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.727558 | controller |     - '          exit 1'
2026-03-14 09:56:26.727562 | controller |     - '        fi'
2026-03-14 09:56:26.727566 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.727571 | controller |     - '        '
2026-03-14 09:56:26.727575 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.727579 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.727584 | controller |       | base64 -d)'
2026-03-14 09:56:26.727588 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.727592 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.727614 | controller |     - '          exit 1'
2026-03-14 09:56:26.727619 | controller |     - '        fi'
2026-03-14 09:56:26.727623 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.727627 | controller |     - '        '
2026-03-14 09:56:26.727632 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.727636 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.727640 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.727644 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.727649 | controller |       --timeout=300s'
2026-03-14 09:56:26.727653 | controller |     - '        '
2026-03-14 09:56:26.727657 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.727662 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.727668 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.727672 | controller |     - '        '
2026-03-14 09:56:26.727676 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.727681 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.727685 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.727689 | controller |     - '        '
2026-03-14 09:56:26.727700 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.727705 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.727709 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.727714 | controller |       available"'
2026-03-14 09:56:26.727718 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.727722 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.727729 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.727733 | controller |     - '          exit 1'
2026-03-14 09:56:26.727737 | controller |     - '        fi'
2026-03-14 09:56:26.727742 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.727746 | controller |     - '        '
2026-03-14 09:56:26.727750 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.727754 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.727759 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.727763 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.727767 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.727772 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.727776 | controller |       -- \'
2026-03-14 09:56:26.727780 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.727784 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.727789 | controller |     - '    logger.go:42: 09:43:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.727793 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.727797 | controller |     - '    logger.go:42: 09:43:27 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.727802 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.727806 | controller |       found'
2026-03-14 09:56:26.727810 | controller |     - '    logger.go:42: 09:43:28 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.727814 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.727819 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.727823 | controller |     - '        '
2026-03-14 09:56:26.727827 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.727833 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.727838 | controller |     - '        '
2026-03-14 09:56:26.727842 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.727846 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.727850 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.727855 | controller |     - '          exit 1'
2026-03-14 09:56:26.727861 | controller |     - '        fi'
2026-03-14 09:56:26.727865 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.727869 | controller |     - '        '
2026-03-14 09:56:26.727874 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.727878 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.727882 | controller |       | base64 -d)'
2026-03-14 09:56:26.727887 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.727891 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.727895 | controller |     - '          exit 1'
2026-03-14 09:56:26.727899 | controller |     - '        fi'
2026-03-14 09:56:26.727904 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.727908 | controller |     - '        '
2026-03-14 09:56:26.727912 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.727917 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.727921 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.727925 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.727929 | controller |       --timeout=300s'
2026-03-14 09:56:26.727934 | controller |     - '        '
2026-03-14 09:56:26.727938 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.727942 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.727946 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.727951 | controller |     - '        '
2026-03-14 09:56:26.727955 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.727959 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.727964 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.727968 | controller |     - '        '
2026-03-14 09:56:26.727978 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.727982 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.727987 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.727991 | controller |       available"'
2026-03-14 09:56:26.727995 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.728000 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.728004 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.728008 | controller |     - '          exit 1'
2026-03-14 09:56:26.728013 | controller |     - '        fi'
2026-03-14 09:56:26.728017 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.728021 | controller |     - '        '
2026-03-14 09:56:26.728026 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.728030 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.728034 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.728038 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.728043 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.728047 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.728053 | controller |       -- \'
2026-03-14 09:56:26.728057 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.728062 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.728066 | controller |     - '    logger.go:42: 09:43:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.728070 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.728075 | controller |     - '    logger.go:42: 09:43:28 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.728079 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.728083 | controller |       found'
2026-03-14 09:56:26.728087 | controller |     - '    logger.go:42: 09:43:29 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.728092 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.728096 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.728100 | controller |     - '        '
2026-03-14 09:56:26.728105 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.728109 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.728113 | controller |     - '        '
2026-03-14 09:56:26.728117 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.728122 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.728126 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.728130 | controller |     - '          exit 1'
2026-03-14 09:56:26.728135 | controller |     - '        fi'
2026-03-14 09:56:26.728139 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.728143 | controller |     - '        '
2026-03-14 09:56:26.728148 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.728152 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.728156 | controller |       | base64 -d)'
2026-03-14 09:56:26.728160 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.728165 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.728169 | controller |     - '          exit 1'
2026-03-14 09:56:26.728173 | controller |     - '        fi'
2026-03-14 09:56:26.728178 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.728182 | controller |     - '        '
2026-03-14 09:56:26.728186 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.728190 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.728195 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.728199 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.728203 | controller |       --timeout=300s'
2026-03-14 09:56:26.728208 | controller |     - '        '
2026-03-14 09:56:26.728212 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.728216 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.728221 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.728225 | controller |     - '        '
2026-03-14 09:56:26.728229 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.728233 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.728238 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.728245 | controller |     - '        '
2026-03-14 09:56:26.728250 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.728260 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.728264 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.728269 | controller |       available"'
2026-03-14 09:56:26.728273 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.728277 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.728282 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.728286 | controller |     - '          exit 1'
2026-03-14 09:56:26.728291 | controller |     - '        fi'
2026-03-14 09:56:26.728295 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.728299 | controller |     - '        '
2026-03-14 09:56:26.728304 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.728308 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.728312 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.728316 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.728321 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.728325 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.728331 | controller |       -- \'
2026-03-14 09:56:26.728336 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.728340 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.728344 | controller |     - '    logger.go:42: 09:43:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.728349 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.728353 | controller |     - '    logger.go:42: 09:43:29 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.728357 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.728362 | controller |       found'
2026-03-14 09:56:26.728366 | controller |     - '    logger.go:42: 09:43:30 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.728370 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.728375 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.728379 | controller |     - '        '
2026-03-14 09:56:26.728383 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.728388 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.728394 | controller |     - '        '
2026-03-14 09:56:26.728398 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.728402 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.728407 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.728411 | controller |     - '          exit 1'
2026-03-14 09:56:26.728415 | controller |     - '        fi'
2026-03-14 09:56:26.728420 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.728424 | controller |     - '        '
2026-03-14 09:56:26.728428 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.728432 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.728437 | controller |       | base64 -d)'
2026-03-14 09:56:26.728441 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.728445 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.728451 | controller |     - '          exit 1'
2026-03-14 09:56:26.728456 | controller |     - '        fi'
2026-03-14 09:56:26.728460 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.728464 | controller |     - '        '
2026-03-14 09:56:26.728469 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.728473 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.728477 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.728482 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.728486 | controller |       --timeout=300s'
2026-03-14 09:56:26.728490 | controller |     - '        '
2026-03-14 09:56:26.728495 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.728501 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.728509 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.728515 | controller |     - '        '
2026-03-14 09:56:26.728521 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.728527 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.728533 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.728539 | controller |     - '        '
2026-03-14 09:56:26.728544 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.728548 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.728559 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.728564 | controller |       available"'
2026-03-14 09:56:26.728571 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.728575 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.728579 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.728584 | controller |     - '          exit 1'
2026-03-14 09:56:26.728588 | controller |     - '        fi'
2026-03-14 09:56:26.728592 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.728610 | controller |     - '        '
2026-03-14 09:56:26.728615 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.728619 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.728623 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.728628 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.728632 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.728636 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.728641 | controller |       -- \'
2026-03-14 09:56:26.728645 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.728649 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.728653 | controller |     - '    logger.go:42: 09:43:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.728658 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.728662 | controller |     - '    logger.go:42: 09:43:30 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.728669 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.728673 | controller |       found'
2026-03-14 09:56:26.728677 | controller |     - '    logger.go:42: 09:43:31 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.728681 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.728686 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.728690 | controller |     - '        '
2026-03-14 09:56:26.728694 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.728699 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.728703 | controller |     - '        '
2026-03-14 09:56:26.728707 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.728711 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.728716 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.728720 | controller |     - '          exit 1'
2026-03-14 09:56:26.728724 | controller |     - '        fi'
2026-03-14 09:56:26.728730 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.728734 | controller |     - '        '
2026-03-14 09:56:26.728738 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.728742 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.728747 | controller |       | base64 -d)'
2026-03-14 09:56:26.728751 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.728755 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.728760 | controller |     - '          exit 1'
2026-03-14 09:56:26.728764 | controller |     - '        fi'
2026-03-14 09:56:26.728768 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.728772 | controller |     - '        '
2026-03-14 09:56:26.728777 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.728781 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.728785 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.728789 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.728794 | controller |       --timeout=300s'
2026-03-14 09:56:26.728798 | controller |     - '        '
2026-03-14 09:56:26.728802 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.728807 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.728811 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.728815 | controller |     - '        '
2026-03-14 09:56:26.728819 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.728824 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.728828 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.728832 | controller |     - '        '
2026-03-14 09:56:26.728836 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.728841 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.728852 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.728856 | controller |       available"'
2026-03-14 09:56:26.728860 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.728865 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.728871 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.728875 | controller |     - '          exit 1'
2026-03-14 09:56:26.728879 | controller |     - '        fi'
2026-03-14 09:56:26.728884 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.728888 | controller |     - '        '
2026-03-14 09:56:26.728892 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.728897 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.728901 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.728905 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.728909 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.728914 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.728918 | controller |       -- \'
2026-03-14 09:56:26.728922 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.728927 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.728931 | controller |     - '    logger.go:42: 09:43:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.728935 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.728939 | controller |     - '    logger.go:42: 09:43:31 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.728944 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.728948 | controller |       found'
2026-03-14 09:56:26.728952 | controller |     - '    logger.go:42: 09:43:32 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.728956 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.728961 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.728965 | controller |     - '        '
2026-03-14 09:56:26.728969 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.728974 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.728978 | controller |     - '        '
2026-03-14 09:56:26.728982 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.728989 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.728993 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.728997 | controller |     - '          exit 1'
2026-03-14 09:56:26.729002 | controller |     - '        fi'
2026-03-14 09:56:26.729006 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.729010 | controller |     - '        '
2026-03-14 09:56:26.729014 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.729020 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.729025 | controller |       | base64 -d)'
2026-03-14 09:56:26.729029 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.729033 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.729037 | controller |     - '          exit 1'
2026-03-14 09:56:26.729042 | controller |     - '        fi'
2026-03-14 09:56:26.729046 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.729050 | controller |     - '        '
2026-03-14 09:56:26.729055 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.729059 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.729063 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.729069 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.729073 | controller |       --timeout=300s'
2026-03-14 09:56:26.729077 | controller |     - '        '
2026-03-14 09:56:26.729082 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.729086 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.729090 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.729095 | controller |     - '        '
2026-03-14 09:56:26.729099 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.729103 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.729107 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.729112 | controller |     - '        '
2026-03-14 09:56:26.729117 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.729123 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.729129 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.729135 | controller |       available"'
2026-03-14 09:56:26.729149 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.729154 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.729158 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.729163 | controller |     - '          exit 1'
2026-03-14 09:56:26.729167 | controller |     - '        fi'
2026-03-14 09:56:26.729171 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.729176 | controller |     - '        '
2026-03-14 09:56:26.729180 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.729184 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.729188 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.729193 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.729197 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.729201 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.729206 | controller |       -- \'
2026-03-14 09:56:26.729210 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.729214 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.729219 | controller |     - '    logger.go:42: 09:43:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.729223 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.729227 | controller |     - '    logger.go:42: 09:43:32 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.729231 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.729236 | controller |       found'
2026-03-14 09:56:26.729240 | controller |     - '    logger.go:42: 09:43:33 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.729244 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.729249 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.729253 | controller |     - '        '
2026-03-14 09:56:26.729257 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.729263 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.729268 | controller |     - '        '
2026-03-14 09:56:26.729272 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.729277 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.729281 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.729285 | controller |     - '          exit 1'
2026-03-14 09:56:26.729290 | controller |     - '        fi'
2026-03-14 09:56:26.729294 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.729298 | controller |     - '        '
2026-03-14 09:56:26.729302 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.729307 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.729311 | controller |       | base64 -d)'
2026-03-14 09:56:26.729315 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.729320 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.729324 | controller |     - '          exit 1'
2026-03-14 09:56:26.729329 | controller |     - '        fi'
2026-03-14 09:56:26.729333 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.729337 | controller |     - '        '
2026-03-14 09:56:26.729342 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.729346 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.729350 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.729356 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.729361 | controller |       --timeout=300s'
2026-03-14 09:56:26.729365 | controller |     - '        '
2026-03-14 09:56:26.729369 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.729374 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.729378 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.729383 | controller |     - '        '
2026-03-14 09:56:26.729387 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.729391 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.729395 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.729400 | controller |     - '        '
2026-03-14 09:56:26.729404 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.729408 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.729413 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.729417 | controller |       available"'
2026-03-14 09:56:26.729427 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.729432 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.729436 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.729441 | controller |     - '          exit 1'
2026-03-14 09:56:26.729445 | controller |     - '        fi'
2026-03-14 09:56:26.729449 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.729454 | controller |     - '        '
2026-03-14 09:56:26.729458 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.729462 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.729469 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.729475 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.729479 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.729484 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.729488 | controller |       -- \'
2026-03-14 09:56:26.729492 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.729496 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.729501 | controller |     - '    logger.go:42: 09:43:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.729505 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.729509 | controller |     - '    logger.go:42: 09:43:33 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.729513 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.729518 | controller |       found'
2026-03-14 09:56:26.729522 | controller |     - '    logger.go:42: 09:43:34 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.729526 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.729530 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.729535 | controller |     - '        '
2026-03-14 09:56:26.729539 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.729543 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.729547 | controller |     - '        '
2026-03-14 09:56:26.729552 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.729556 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.729560 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.729564 | controller |     - '          exit 1'
2026-03-14 09:56:26.729569 | controller |     - '        fi'
2026-03-14 09:56:26.729573 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.729577 | controller |     - '        '
2026-03-14 09:56:26.729582 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.729586 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.729590 | controller |       | base64 -d)'
2026-03-14 09:56:26.729605 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.729612 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.729616 | controller |     - '          exit 1'
2026-03-14 09:56:26.729620 | controller |     - '        fi'
2026-03-14 09:56:26.729625 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.729633 | controller |     - '        '
2026-03-14 09:56:26.729637 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.729641 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.729647 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.729651 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.729656 | controller |       --timeout=300s'
2026-03-14 09:56:26.729660 | controller |     - '        '
2026-03-14 09:56:26.729664 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.729671 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.729675 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.729681 | controller |     - '        '
2026-03-14 09:56:26.729685 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.729690 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.729694 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.729698 | controller |     - '        '
2026-03-14 09:56:26.729702 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.729707 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.729711 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.729715 | controller |       available"'
2026-03-14 09:56:26.729720 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.729731 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.729735 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.729740 | controller |     - '          exit 1'
2026-03-14 09:56:26.729744 | controller |     - '        fi'
2026-03-14 09:56:26.729748 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.729753 | controller |     - '        '
2026-03-14 09:56:26.729757 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.729761 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.729766 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.729771 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.729777 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.729783 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.729789 | controller |       -- \'
2026-03-14 09:56:26.729795 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.729800 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.729806 | controller |     - '    logger.go:42: 09:43:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.729812 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.729818 | controller |     - '    logger.go:42: 09:43:35 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.729824 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.729829 | controller |       found'
2026-03-14 09:56:26.729835 | controller |     - '    logger.go:42: 09:43:36 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.729842 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.729847 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.729852 | controller |     - '        '
2026-03-14 09:56:26.729856 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.729860 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.729865 | controller |     - '        '
2026-03-14 09:56:26.729869 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.729873 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.729878 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.729882 | controller |     - '          exit 1'
2026-03-14 09:56:26.729886 | controller |     - '        fi'
2026-03-14 09:56:26.729891 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.729898 | controller |     - '        '
2026-03-14 09:56:26.729902 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.729906 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.729911 | controller |       | base64 -d)'
2026-03-14 09:56:26.729915 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.729920 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.729924 | controller |     - '          exit 1'
2026-03-14 09:56:26.729928 | controller |     - '        fi'
2026-03-14 09:56:26.729933 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.729937 | controller |     - '        '
2026-03-14 09:56:26.729941 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.729946 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.729950 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.729954 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.729959 | controller |       --timeout=300s'
2026-03-14 09:56:26.729963 | controller |     - '        '
2026-03-14 09:56:26.729967 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.729972 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.729976 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.729980 | controller |     - '        '
2026-03-14 09:56:26.729985 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.729989 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.729993 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.729998 | controller |     - '        '
2026-03-14 09:56:26.730002 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.730006 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.730011 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.730015 | controller |       available"'
2026-03-14 09:56:26.730019 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.730024 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.730036 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.730041 | controller |     - '          exit 1'
2026-03-14 09:56:26.730045 | controller |     - '        fi'
2026-03-14 09:56:26.730050 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.730054 | controller |     - '        '
2026-03-14 09:56:26.730059 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.730063 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.730067 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.730072 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.730076 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.730080 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.730085 | controller |       -- \'
2026-03-14 09:56:26.730092 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.730096 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.730100 | controller |     - '    logger.go:42: 09:43:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.730105 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.730109 | controller |     - '    logger.go:42: 09:43:36 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.730114 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.730118 | controller |       found'
2026-03-14 09:56:26.730122 | controller |     - '    logger.go:42: 09:43:37 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.730127 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.730131 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.730135 | controller |     - '        '
2026-03-14 09:56:26.730140 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.730144 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.730149 | controller |     - '        '
2026-03-14 09:56:26.730153 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.730157 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.730164 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.730169 | controller |     - '          exit 1'
2026-03-14 09:56:26.730173 | controller |     - '        fi'
2026-03-14 09:56:26.730178 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.730182 | controller |     - '        '
2026-03-14 09:56:26.730186 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.730191 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.730195 | controller |       | base64 -d)'
2026-03-14 09:56:26.730200 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.730204 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.730208 | controller |     - '          exit 1'
2026-03-14 09:56:26.730213 | controller |     - '        fi'
2026-03-14 09:56:26.730217 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.730221 | controller |     - '        '
2026-03-14 09:56:26.730226 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.730230 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.730235 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.730241 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.730245 | controller |       --timeout=300s'
2026-03-14 09:56:26.730249 | controller |     - '        '
2026-03-14 09:56:26.730254 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.730258 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.730263 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.730267 | controller |     - '        '
2026-03-14 09:56:26.730271 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.730276 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.730280 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.730285 | controller |     - '        '
2026-03-14 09:56:26.730291 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.730296 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.730300 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.730304 | controller |       available"'
2026-03-14 09:56:26.730309 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.730313 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.730319 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.730330 | controller |     - '          exit 1'
2026-03-14 09:56:26.730334 | controller |     - '        fi'
2026-03-14 09:56:26.730339 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.730343 | controller |     - '        '
2026-03-14 09:56:26.730348 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.730352 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.730357 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.730361 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.730365 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.730370 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.730374 | controller |       -- \'
2026-03-14 09:56:26.730379 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.730383 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.730387 | controller |     - '    logger.go:42: 09:43:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.730392 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.730396 | controller |     - '    logger.go:42: 09:43:37 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.730401 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.730405 | controller |       found'
2026-03-14 09:56:26.730410 | controller |     - '    logger.go:42: 09:43:38 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.730414 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.730419 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.730423 | controller |     - '        '
2026-03-14 09:56:26.730428 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.730432 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.730437 | controller |     - '        '
2026-03-14 09:56:26.730441 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.730445 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.730450 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.730454 | controller |     - '          exit 1'
2026-03-14 09:56:26.730459 | controller |     - '        fi'
2026-03-14 09:56:26.730463 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.730467 | controller |     - '        '
2026-03-14 09:56:26.730472 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.730476 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.730480 | controller |       | base64 -d)'
2026-03-14 09:56:26.730485 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.730489 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.730495 | controller |     - '          exit 1'
2026-03-14 09:56:26.730499 | controller |     - '        fi'
2026-03-14 09:56:26.730504 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.730508 | controller |     - '        '
2026-03-14 09:56:26.730513 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.730517 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.730521 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.730526 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.730530 | controller |       --timeout=300s'
2026-03-14 09:56:26.730535 | controller |     - '        '
2026-03-14 09:56:26.730539 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.730543 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.730548 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.730554 | controller |     - '        '
2026-03-14 09:56:26.730558 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.730563 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.730567 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.730572 | controller |     - '        '
2026-03-14 09:56:26.730576 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.730580 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.730585 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.730589 | controller |       available"'
2026-03-14 09:56:26.730604 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.730611 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.730616 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.730620 | controller |     - '          exit 1'
2026-03-14 09:56:26.730625 | controller |     - '        fi'
2026-03-14 09:56:26.730636 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.730640 | controller |     - '        '
2026-03-14 09:56:26.730645 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.730649 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.730654 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.730658 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.730662 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.730667 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.730671 | controller |       -- \'
2026-03-14 09:56:26.730676 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.730680 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.730684 | controller |     - '    logger.go:42: 09:43:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.730689 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.730693 | controller |     - '    logger.go:42: 09:43:38 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.730700 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.730704 | controller |       found'
2026-03-14 09:56:26.730708 | controller |     - '    logger.go:42: 09:43:39 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.730713 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.730717 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.730722 | controller |     - '        '
2026-03-14 09:56:26.730726 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.730730 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.730735 | controller |     - '        '
2026-03-14 09:56:26.730739 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.730744 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.730748 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.730752 | controller |     - '          exit 1'
2026-03-14 09:56:26.730757 | controller |     - '        fi'
2026-03-14 09:56:26.730763 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.730767 | controller |     - '        '
2026-03-14 09:56:26.730772 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.730776 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.730780 | controller |       | base64 -d)'
2026-03-14 09:56:26.730785 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.730789 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.730794 | controller |     - '          exit 1'
2026-03-14 09:56:26.730798 | controller |     - '        fi'
2026-03-14 09:56:26.730802 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.730807 | controller |     - '        '
2026-03-14 09:56:26.730811 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.730815 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.730820 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.730824 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.730830 | controller |       --timeout=300s'
2026-03-14 09:56:26.730834 | controller |     - '        '
2026-03-14 09:56:26.730839 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.730843 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.730848 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.730852 | controller |     - '        '
2026-03-14 09:56:26.730856 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.730861 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.730865 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.730869 | controller |     - '        '
2026-03-14 09:56:26.730874 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.730878 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.730882 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.730887 | controller |       available"'
2026-03-14 09:56:26.730891 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.730895 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.730900 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.730907 | controller |     - '          exit 1'
2026-03-14 09:56:26.730912 | controller |     - '        fi'
2026-03-14 09:56:26.730925 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.730930 | controller |     - '        '
2026-03-14 09:56:26.730935 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.730939 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.730944 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.730948 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.730952 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.730957 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.730961 | controller |       -- \'
2026-03-14 09:56:26.730966 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.730970 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.730974 | controller |     - '    logger.go:42: 09:43:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.730979 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.730983 | controller |     - '    logger.go:42: 09:43:39 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.730987 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.730992 | controller |       found'
2026-03-14 09:56:26.730996 | controller |     - '    logger.go:42: 09:43:40 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.731018 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.731024 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.731028 | controller |     - '        '
2026-03-14 09:56:26.731033 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.731037 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.731041 | controller |     - '        '
2026-03-14 09:56:26.731046 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.731050 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.731054 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.731059 | controller |     - '          exit 1'
2026-03-14 09:56:26.731063 | controller |     - '        fi'
2026-03-14 09:56:26.731068 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.731072 | controller |     - '        '
2026-03-14 09:56:26.731076 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.731083 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.731091 | controller |       | base64 -d)'
2026-03-14 09:56:26.731096 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.731100 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.731104 | controller |     - '          exit 1'
2026-03-14 09:56:26.731109 | controller |     - '        fi'
2026-03-14 09:56:26.731113 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.731118 | controller |     - '        '
2026-03-14 09:56:26.731122 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.731126 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.731131 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.731135 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.731141 | controller |       --timeout=300s'
2026-03-14 09:56:26.731146 | controller |     - '        '
2026-03-14 09:56:26.731150 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.731155 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.731159 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.731164 | controller |     - '        '
2026-03-14 09:56:26.731168 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.731172 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.731177 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.731181 | controller |     - '        '
2026-03-14 09:56:26.731185 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.731190 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.731194 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.731199 | controller |       available"'
2026-03-14 09:56:26.731203 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.731207 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.731212 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.731216 | controller |     - '          exit 1'
2026-03-14 09:56:26.731220 | controller |     - '        fi'
2026-03-14 09:56:26.731225 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.731229 | controller |     - '        '
2026-03-14 09:56:26.731240 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.731245 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.731249 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.731253 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.731258 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.731262 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.731267 | controller |       -- \'
2026-03-14 09:56:26.731271 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.731275 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.731280 | controller |     - '    logger.go:42: 09:43:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.731284 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.731289 | controller |     - '    logger.go:42: 09:43:41 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.731293 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.731297 | controller |       found'
2026-03-14 09:56:26.731302 | controller |     - '    logger.go:42: 09:43:42 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.731306 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.731310 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.731315 | controller |     - '        '
2026-03-14 09:56:26.731319 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.731323 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.731330 | controller |     - '        '
2026-03-14 09:56:26.731334 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.731338 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.731343 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.731347 | controller |     - '          exit 1'
2026-03-14 09:56:26.731352 | controller |     - '        fi'
2026-03-14 09:56:26.731356 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.731360 | controller |     - '        '
2026-03-14 09:56:26.731365 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.731369 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.731373 | controller |       | base64 -d)'
2026-03-14 09:56:26.731378 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.731382 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.731386 | controller |     - '          exit 1'
2026-03-14 09:56:26.731391 | controller |     - '        fi'
2026-03-14 09:56:26.731395 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.731399 | controller |     - '        '
2026-03-14 09:56:26.731404 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.731408 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.731412 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.731417 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.731421 | controller |       --timeout=300s'
2026-03-14 09:56:26.731426 | controller |     - '        '
2026-03-14 09:56:26.731430 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.731434 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.731439 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.731443 | controller |     - '        '
2026-03-14 09:56:26.731448 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.731452 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.731456 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.731461 | controller |     - '        '
2026-03-14 09:56:26.731465 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.731479 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.731483 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.731488 | controller |       available"'
2026-03-14 09:56:26.731492 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.731496 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.731501 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.731505 | controller |     - '          exit 1'
2026-03-14 09:56:26.731509 | controller |     - '        fi'
2026-03-14 09:56:26.731513 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.731518 | controller |     - '        '
2026-03-14 09:56:26.731528 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.731533 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.731537 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.731544 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.731548 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.731554 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.731558 | controller |       -- \'
2026-03-14 09:56:26.731563 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.731567 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.731571 | controller |     - '    logger.go:42: 09:43:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.731575 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.731580 | controller |     - '    logger.go:42: 09:43:42 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.731584 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.731588 | controller |       found'
2026-03-14 09:56:26.731593 | controller |     - '    logger.go:42: 09:43:43 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.731610 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.731614 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.731618 | controller |     - '        '
2026-03-14 09:56:26.731623 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.731627 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.731631 | controller |     - '        '
2026-03-14 09:56:26.731635 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.731640 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.731644 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.731648 | controller |     - '          exit 1'
2026-03-14 09:56:26.731652 | controller |     - '        fi'
2026-03-14 09:56:26.731657 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.731661 | controller |     - '        '
2026-03-14 09:56:26.731665 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.731671 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.731676 | controller |       | base64 -d)'
2026-03-14 09:56:26.731680 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.731684 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.731689 | controller |     - '          exit 1'
2026-03-14 09:56:26.731693 | controller |     - '        fi'
2026-03-14 09:56:26.731697 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.731701 | controller |     - '        '
2026-03-14 09:56:26.731706 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.731710 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.731714 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.731718 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.731723 | controller |       --timeout=300s'
2026-03-14 09:56:26.731765 | controller |     - '        '
2026-03-14 09:56:26.731775 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.731781 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.731786 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.731823 | controller |     - '        '
2026-03-14 09:56:26.731829 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.731834 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.731838 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.731842 | controller |     - '        '
2026-03-14 09:56:26.731847 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.731851 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.731855 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.731860 | controller |       available"'
2026-03-14 09:56:26.731864 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.731873 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.731879 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.731885 | controller |     - '          exit 1'
2026-03-14 09:56:26.731890 | controller |     - '        fi'
2026-03-14 09:56:26.731897 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.731901 | controller |     - '        '
2026-03-14 09:56:26.731905 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.731918 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.731922 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.731927 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.731931 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.731935 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.731972 | controller |       -- \'
2026-03-14 09:56:26.732009 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.732015 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.732021 | controller |     - '    logger.go:42: 09:43:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.732026 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.732031 | controller |     - '    logger.go:42: 09:43:43 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.732037 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.732042 | controller |       found'
2026-03-14 09:56:26.732048 | controller |     - '    logger.go:42: 09:43:44 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.732053 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.732059 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.732065 | controller |     - '        '
2026-03-14 09:56:26.732071 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.732076 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.732082 | controller |     - '        '
2026-03-14 09:56:26.732088 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.732094 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.732099 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.732105 | controller |     - '          exit 1'
2026-03-14 09:56:26.732110 | controller |     - '        fi'
2026-03-14 09:56:26.732116 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.732123 | controller |     - '        '
2026-03-14 09:56:26.732128 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.732132 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.732136 | controller |       | base64 -d)'
2026-03-14 09:56:26.732140 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.732145 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.732149 | controller |     - '          exit 1'
2026-03-14 09:56:26.732153 | controller |     - '        fi'
2026-03-14 09:56:26.732157 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.732192 | controller |     - '        '
2026-03-14 09:56:26.732197 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.732201 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.732205 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.732210 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.732214 | controller |       --timeout=300s'
2026-03-14 09:56:26.732218 | controller |     - '        '
2026-03-14 09:56:26.732222 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.732227 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.732231 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.732235 | controller |     - '        '
2026-03-14 09:56:26.732240 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.732244 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.732248 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.732253 | controller |     - '        '
2026-03-14 09:56:26.732257 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.732261 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.732265 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.732280 | controller |       available"'
2026-03-14 09:56:26.732288 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.732293 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.732297 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.732301 | controller |     - '          exit 1'
2026-03-14 09:56:26.732306 | controller |     - '        fi'
2026-03-14 09:56:26.732310 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.732314 | controller |     - '        '
2026-03-14 09:56:26.732318 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.732323 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.732336 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.732341 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.732345 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.732349 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.732354 | controller |       -- \'
2026-03-14 09:56:26.732358 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.732365 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.732369 | controller |     - '    logger.go:42: 09:43:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.732373 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.732378 | controller |     - '    logger.go:42: 09:43:44 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.732382 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.732386 | controller |       found'
2026-03-14 09:56:26.732390 | controller |     - '    logger.go:42: 09:43:45 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.732395 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.732399 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.732403 | controller |     - '        '
2026-03-14 09:56:26.732408 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.732412 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.732416 | controller |     - '        '
2026-03-14 09:56:26.732421 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.732425 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.732429 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.732433 | controller |     - '          exit 1'
2026-03-14 09:56:26.732438 | controller |     - '        fi'
2026-03-14 09:56:26.732442 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.732446 | controller |     - '        '
2026-03-14 09:56:26.732450 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.732455 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.732459 | controller |       | base64 -d)'
2026-03-14 09:56:26.732463 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.732468 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.732472 | controller |     - '          exit 1'
2026-03-14 09:56:26.732476 | controller |     - '        fi'
2026-03-14 09:56:26.732480 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.732485 | controller |     - '        '
2026-03-14 09:56:26.732489 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.732493 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.732498 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.732502 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.732506 | controller |       --timeout=300s'
2026-03-14 09:56:26.732510 | controller |     - '        '
2026-03-14 09:56:26.732515 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.732519 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.732523 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.732528 | controller |     - '        '
2026-03-14 09:56:26.732532 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.732536 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.732540 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.732545 | controller |     - '        '
2026-03-14 09:56:26.732549 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.732556 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.732561 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.732565 | controller |       available"'
2026-03-14 09:56:26.732569 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.732574 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.732578 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.732582 | controller |     - '          exit 1'
2026-03-14 09:56:26.732586 | controller |     - '        fi'
2026-03-14 09:56:26.732591 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.732606 | controller |     - '        '
2026-03-14 09:56:26.732613 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.732618 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.732628 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.732633 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.732637 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.732641 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.732665 | controller |       -- \'
2026-03-14 09:56:26.732669 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.732674 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.732678 | controller |     - '    logger.go:42: 09:43:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.732682 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.732687 | controller |     - '    logger.go:42: 09:43:46 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.732691 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.732695 | controller |       found'
2026-03-14 09:56:26.732699 | controller |     - '    logger.go:42: 09:43:47 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.732704 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.732708 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.732712 | controller |     - '        '
2026-03-14 09:56:26.732717 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.732721 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.732725 | controller |     - '        '
2026-03-14 09:56:26.732729 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.732734 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.732738 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.732742 | controller |     - '          exit 1'
2026-03-14 09:56:26.732746 | controller |     - '        fi'
2026-03-14 09:56:26.732751 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.732755 | controller |     - '        '
2026-03-14 09:56:26.732759 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.732763 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.732768 | controller |       | base64 -d)'
2026-03-14 09:56:26.732772 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.732776 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.732781 | controller |     - '          exit 1'
2026-03-14 09:56:26.732785 | controller |     - '        fi'
2026-03-14 09:56:26.732791 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.732795 | controller |     - '        '
2026-03-14 09:56:26.732800 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.732804 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.732808 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.732812 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.732817 | controller |       --timeout=300s'
2026-03-14 09:56:26.732821 | controller |     - '        '
2026-03-14 09:56:26.732825 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.732830 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.732834 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.732838 | controller |     - '        '
2026-03-14 09:56:26.732843 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.732847 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.732853 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.732858 | controller |     - '        '
2026-03-14 09:56:26.732862 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.732866 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.732879 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.732884 | controller |       available"'
2026-03-14 09:56:26.732888 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.732893 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.732897 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.732901 | controller |     - '          exit 1'
2026-03-14 09:56:26.732905 | controller |     - '        fi'
2026-03-14 09:56:26.732910 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.732923 | controller |     - '        '
2026-03-14 09:56:26.732928 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.732936 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.732953 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.732958 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.732962 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.732966 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.732971 | controller |       -- \'
2026-03-14 09:56:26.732975 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.732979 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.732984 | controller |     - '    logger.go:42: 09:43:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.732988 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.732992 | controller |     - '    logger.go:42: 09:43:47 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.732996 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.733003 | controller |       found'
2026-03-14 09:56:26.733007 | controller |     - '    logger.go:42: 09:43:48 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.733011 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.733016 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.733020 | controller |     - '        '
2026-03-14 09:56:26.733024 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.733028 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.733033 | controller |     - '        '
2026-03-14 09:56:26.733037 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.733041 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.733047 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.733053 | controller |     - '          exit 1'
2026-03-14 09:56:26.733094 | controller |     - '        fi'
2026-03-14 09:56:26.733099 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.733104 | controller |     - '        '
2026-03-14 09:56:26.733108 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.733112 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.733116 | controller |       | base64 -d)'
2026-03-14 09:56:26.733121 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.733125 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.733129 | controller |     - '          exit 1'
2026-03-14 09:56:26.733133 | controller |     - '        fi'
2026-03-14 09:56:26.733138 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.733142 | controller |     - '        '
2026-03-14 09:56:26.733146 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.733151 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.733155 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.733159 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.733163 | controller |       --timeout=300s'
2026-03-14 09:56:26.733168 | controller |     - '        '
2026-03-14 09:56:26.733174 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.733179 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.733183 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.733187 | controller |     - '        '
2026-03-14 09:56:26.733192 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.733196 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.733200 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.733205 | controller |     - '        '
2026-03-14 09:56:26.733209 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.733213 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.733217 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.733222 | controller |       available"'
2026-03-14 09:56:26.733226 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.733230 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.733236 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.733242 | controller |     - '          exit 1'
2026-03-14 09:56:26.733247 | controller |     - '        fi'
2026-03-14 09:56:26.733251 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.733255 | controller |     - '        '
2026-03-14 09:56:26.733260 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.733264 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.733268 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.733280 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.733285 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.733289 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.733294 | controller |       -- \'
2026-03-14 09:56:26.733304 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.733308 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.733313 | controller |     - '    logger.go:42: 09:43:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.733317 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.733321 | controller |     - '    logger.go:42: 09:43:48 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.733326 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.733330 | controller |       found'
2026-03-14 09:56:26.733334 | controller |     - '    logger.go:42: 09:43:49 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.733338 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.733343 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.733347 | controller |     - '        '
2026-03-14 09:56:26.733351 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.733355 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.733360 | controller |     - '        '
2026-03-14 09:56:26.733364 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.733368 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.733373 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.733377 | controller |     - '          exit 1'
2026-03-14 09:56:26.733381 | controller |     - '        fi'
2026-03-14 09:56:26.733385 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.733390 | controller |     - '        '
2026-03-14 09:56:26.733394 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.733398 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.733402 | controller |       | base64 -d)'
2026-03-14 09:56:26.733407 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.733411 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.733415 | controller |     - '          exit 1'
2026-03-14 09:56:26.733419 | controller |     - '        fi'
2026-03-14 09:56:26.733424 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.733428 | controller |     - '        '
2026-03-14 09:56:26.733432 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.733436 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.733441 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.733445 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.733452 | controller |       --timeout=300s'
2026-03-14 09:56:26.733456 | controller |     - '        '
2026-03-14 09:56:26.733461 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.733465 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.733469 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.733474 | controller |     - '        '
2026-03-14 09:56:26.733478 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.733482 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.733486 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.733491 | controller |     - '        '
2026-03-14 09:56:26.733495 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.733499 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.733503 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.733508 | controller |       available"'
2026-03-14 09:56:26.733512 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.733516 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.733520 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.733525 | controller |     - '          exit 1'
2026-03-14 09:56:26.733529 | controller |     - '        fi'
2026-03-14 09:56:26.733533 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.733538 | controller |     - '        '
2026-03-14 09:56:26.733542 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.733548 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.733552 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.733556 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.733568 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.733572 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.733577 | controller |       -- \'
2026-03-14 09:56:26.733581 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.733585 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.733590 | controller |     - '    logger.go:42: 09:43:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.733605 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.733613 | controller |     - '    logger.go:42: 09:43:49 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.733617 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.733622 | controller |       found'
2026-03-14 09:56:26.733626 | controller |     - '    logger.go:42: 09:43:50 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.733630 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.733634 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.733639 | controller |     - '        '
2026-03-14 09:56:26.733643 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.733647 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.733654 | controller |     - '        '
2026-03-14 09:56:26.733658 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.733662 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.733667 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.733671 | controller |     - '          exit 1'
2026-03-14 09:56:26.733675 | controller |     - '        fi'
2026-03-14 09:56:26.733680 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.733684 | controller |     - '        '
2026-03-14 09:56:26.733688 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.733692 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.733697 | controller |       | base64 -d)'
2026-03-14 09:56:26.733701 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.733705 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.733710 | controller |     - '          exit 1'
2026-03-14 09:56:26.733714 | controller |     - '        fi'
2026-03-14 09:56:26.733718 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.733723 | controller |     - '        '
2026-03-14 09:56:26.733727 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.733731 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.733735 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.733740 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.733744 | controller |       --timeout=300s'
2026-03-14 09:56:26.733748 | controller |     - '        '
2026-03-14 09:56:26.733753 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.733757 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.733761 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.733766 | controller |     - '        '
2026-03-14 09:56:26.733770 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.733774 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.733778 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.733783 | controller |     - '        '
2026-03-14 09:56:26.733787 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.733791 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.733796 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.733800 | controller |       available"'
2026-03-14 09:56:26.733804 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.733809 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.733813 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.733817 | controller |     - '          exit 1'
2026-03-14 09:56:26.733821 | controller |     - '        fi'
2026-03-14 09:56:26.733826 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.733830 | controller |     - '        '
2026-03-14 09:56:26.733834 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.733839 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.733848 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.733859 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.733871 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.733876 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.733880 | controller |       -- \'
2026-03-14 09:56:26.733884 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.733891 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.733895 | controller |     - '    logger.go:42: 09:43:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.733899 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.733904 | controller |     - '    logger.go:42: 09:43:50 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.733908 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.733912 | controller |       found'
2026-03-14 09:56:26.733917 | controller |     - '    logger.go:42: 09:43:51 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.733921 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.733925 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.733929 | controller |     - '        '
2026-03-14 09:56:26.733934 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.733938 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.733942 | controller |     - '        '
2026-03-14 09:56:26.733946 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.733951 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.733955 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.733959 | controller |     - '          exit 1'
2026-03-14 09:56:26.733964 | controller |     - '        fi'
2026-03-14 09:56:26.733968 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.733972 | controller |     - '        '
2026-03-14 09:56:26.733976 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.733981 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.733985 | controller |       | base64 -d)'
2026-03-14 09:56:26.733989 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.733994 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.733998 | controller |     - '          exit 1'
2026-03-14 09:56:26.734002 | controller |     - '        fi'
2026-03-14 09:56:26.734007 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.734011 | controller |     - '        '
2026-03-14 09:56:26.734015 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.734020 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.734024 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.734028 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.734032 | controller |       --timeout=300s'
2026-03-14 09:56:26.734037 | controller |     - '        '
2026-03-14 09:56:26.734041 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.734047 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.734051 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.734056 | controller |     - '        '
2026-03-14 09:56:26.734060 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.734071 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.734077 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.734084 | controller |     - '        '
2026-03-14 09:56:26.734090 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.734094 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.734099 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.734103 | controller |       available"'
2026-03-14 09:56:26.734107 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.734112 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.734116 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.734120 | controller |     - '          exit 1'
2026-03-14 09:56:26.734124 | controller |     - '        fi'
2026-03-14 09:56:26.734129 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.734133 | controller |     - '        '
2026-03-14 09:56:26.734137 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.734142 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.734146 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.734150 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.734164 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.734169 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.734173 | controller |       -- \'
2026-03-14 09:56:26.734177 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.734182 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.734186 | controller |     - '    logger.go:42: 09:43:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.734190 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.734195 | controller |     - '    logger.go:42: 09:43:51 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.734199 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.734203 | controller |       found'
2026-03-14 09:56:26.734207 | controller |     - '    logger.go:42: 09:43:52 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.734212 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.734218 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.734222 | controller |     - '        '
2026-03-14 09:56:26.734227 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.734231 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.734235 | controller |     - '        '
2026-03-14 09:56:26.734240 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.734244 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.734248 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.734253 | controller |     - '          exit 1'
2026-03-14 09:56:26.734257 | controller |     - '        fi'
2026-03-14 09:56:26.734261 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.734266 | controller |     - '        '
2026-03-14 09:56:26.734270 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.734276 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.734281 | controller |       | base64 -d)'
2026-03-14 09:56:26.734287 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.734291 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.734295 | controller |     - '          exit 1'
2026-03-14 09:56:26.734300 | controller |     - '        fi'
2026-03-14 09:56:26.734304 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.734308 | controller |     - '        '
2026-03-14 09:56:26.734318 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.734323 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.734327 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.734332 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.734336 | controller |       --timeout=300s'
2026-03-14 09:56:26.734340 | controller |     - '        '
2026-03-14 09:56:26.734344 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.734349 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.734353 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.734357 | controller |     - '        '
2026-03-14 09:56:26.734362 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.734366 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.734370 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.734374 | controller |     - '        '
2026-03-14 09:56:26.734379 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.734383 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.734387 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.734391 | controller |       available"'
2026-03-14 09:56:26.734396 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.734400 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.734404 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.734408 | controller |     - '          exit 1'
2026-03-14 09:56:26.734413 | controller |     - '        fi'
2026-03-14 09:56:26.734417 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.734421 | controller |     - '        '
2026-03-14 09:56:26.734425 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.734430 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.734434 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.734438 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.734442 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.734453 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.734458 | controller |       -- \'
2026-03-14 09:56:26.734463 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.734469 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.734473 | controller |     - '    logger.go:42: 09:43:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.734477 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.734482 | controller |     - '    logger.go:42: 09:43:53 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.734486 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.734490 | controller |       found'
2026-03-14 09:56:26.734495 | controller |     - '    logger.go:42: 09:43:54 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.734499 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.734503 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.734507 | controller |     - '        '
2026-03-14 09:56:26.734512 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.734516 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.734520 | controller |     - '        '
2026-03-14 09:56:26.734524 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.734529 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.734533 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.734537 | controller |     - '          exit 1'
2026-03-14 09:56:26.734542 | controller |     - '        fi'
2026-03-14 09:56:26.734546 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.734550 | controller |     - '        '
2026-03-14 09:56:26.734555 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.734561 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.734565 | controller |       | base64 -d)'
2026-03-14 09:56:26.734569 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.734574 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.734578 | controller |     - '          exit 1'
2026-03-14 09:56:26.734582 | controller |     - '        fi'
2026-03-14 09:56:26.734586 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.734591 | controller |     - '        '
2026-03-14 09:56:26.734606 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.734623 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.734627 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.734632 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.734636 | controller |       --timeout=300s'
2026-03-14 09:56:26.734640 | controller |     - '        '
2026-03-14 09:56:26.734645 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.734649 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.734653 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.734658 | controller |     - '        '
2026-03-14 09:56:26.734662 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.734666 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.734671 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.734677 | controller |     - '        '
2026-03-14 09:56:26.734683 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.734689 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.734698 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.734704 | controller |       available"'
2026-03-14 09:56:26.734709 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.734715 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.734720 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.734724 | controller |     - '          exit 1'
2026-03-14 09:56:26.734729 | controller |     - '        fi'
2026-03-14 09:56:26.734733 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.734737 | controller |     - '        '
2026-03-14 09:56:26.734742 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.734746 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.734750 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.734754 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.734759 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.734774 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.734779 | controller |       -- \'
2026-03-14 09:56:26.734783 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.734787 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.734792 | controller |     - '    logger.go:42: 09:43:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.734796 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.734800 | controller |     - '    logger.go:42: 09:43:54 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.734804 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.734809 | controller |       found'
2026-03-14 09:56:26.734813 | controller |     - '    logger.go:42: 09:43:55 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.734823 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.734828 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.734832 | controller |     - '        '
2026-03-14 09:56:26.734836 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.734840 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.734845 | controller |     - '        '
2026-03-14 09:56:26.734849 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.734853 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.734858 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.734862 | controller |     - '          exit 1'
2026-03-14 09:56:26.734866 | controller |     - '        fi'
2026-03-14 09:56:26.734871 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.734875 | controller |     - '        '
2026-03-14 09:56:26.734879 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.734883 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.734888 | controller |       | base64 -d)'
2026-03-14 09:56:26.734892 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.734896 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.734901 | controller |     - '          exit 1'
2026-03-14 09:56:26.734905 | controller |     - '        fi'
2026-03-14 09:56:26.734909 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.734916 | controller |     - '        '
2026-03-14 09:56:26.734920 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.734925 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.734929 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.734933 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.734938 | controller |       --timeout=300s'
2026-03-14 09:56:26.734942 | controller |     - '        '
2026-03-14 09:56:26.734946 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.734950 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.734955 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.734959 | controller |     - '        '
2026-03-14 09:56:26.734963 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.734968 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.734972 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.734978 | controller |     - '        '
2026-03-14 09:56:26.734982 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.734987 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.734991 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.734995 | controller |       available"'
2026-03-14 09:56:26.735014 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.735023 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.735027 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.735031 | controller |     - '          exit 1'
2026-03-14 09:56:26.735036 | controller |     - '        fi'
2026-03-14 09:56:26.735040 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.735044 | controller |     - '        '
2026-03-14 09:56:26.735048 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.735053 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.735057 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.735061 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.735066 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.735070 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.735074 | controller |       -- \'
2026-03-14 09:56:26.735088 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.735095 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.735101 | controller |     - '    logger.go:42: 09:43:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.735107 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.735113 | controller |     - '    logger.go:42: 09:43:55 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.735118 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.735124 | controller |       found'
2026-03-14 09:56:26.735132 | controller |     - '    logger.go:42: 09:43:56 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.735138 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.735143 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.735149 | controller |     - '        '
2026-03-14 09:56:26.735154 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.735158 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.735163 | controller |     - '        '
2026-03-14 09:56:26.735167 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.735171 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.735175 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.735183 | controller |     - '          exit 1'
2026-03-14 09:56:26.735188 | controller |     - '        fi'
2026-03-14 09:56:26.735192 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.735196 | controller |     - '        '
2026-03-14 09:56:26.735201 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.735205 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.735209 | controller |       | base64 -d)'
2026-03-14 09:56:26.735213 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.735218 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.735222 | controller |     - '          exit 1'
2026-03-14 09:56:26.735226 | controller |     - '        fi'
2026-03-14 09:56:26.735231 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.735235 | controller |     - '        '
2026-03-14 09:56:26.735239 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.735244 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.735248 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.735252 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.735256 | controller |       --timeout=300s'
2026-03-14 09:56:26.735261 | controller |     - '        '
2026-03-14 09:56:26.735265 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.735269 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.735274 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.735278 | controller |     - '        '
2026-03-14 09:56:26.735282 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.735286 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.735291 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.735295 | controller |     - '        '
2026-03-14 09:56:26.735299 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.735303 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.735308 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.735312 | controller |       available"'
2026-03-14 09:56:26.735316 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.735321 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.735325 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.735329 | controller |     - '          exit 1'
2026-03-14 09:56:26.735334 | controller |     - '        fi'
2026-03-14 09:56:26.735340 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.735344 | controller |     - '        '
2026-03-14 09:56:26.735348 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.735353 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.735357 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.735363 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.735367 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.735372 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.735376 | controller |       -- \'
2026-03-14 09:56:26.735387 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.735392 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.735397 | controller |     - '    logger.go:42: 09:43:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.735401 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.735405 | controller |     - '    logger.go:42: 09:43:56 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.735410 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.735414 | controller |       found'
2026-03-14 09:56:26.735418 | controller |     - '    logger.go:42: 09:43:57 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.735422 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.735430 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.735434 | controller |     - '        '
2026-03-14 09:56:26.735439 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.735443 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.735447 | controller |     - '        '
2026-03-14 09:56:26.735452 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.735456 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.735460 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.735465 | controller |     - '          exit 1'
2026-03-14 09:56:26.735469 | controller |     - '        fi'
2026-03-14 09:56:26.735473 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.735479 | controller |     - '        '
2026-03-14 09:56:26.735483 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.735488 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.735494 | controller |       | base64 -d)'
2026-03-14 09:56:26.735498 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.735502 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.735506 | controller |     - '          exit 1'
2026-03-14 09:56:26.735511 | controller |     - '        fi'
2026-03-14 09:56:26.735515 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.735519 | controller |     - '        '
2026-03-14 09:56:26.735524 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.735528 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.735532 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.735536 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.735541 | controller |       --timeout=300s'
2026-03-14 09:56:26.735545 | controller |     - '        '
2026-03-14 09:56:26.735551 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.735555 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.735560 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.735564 | controller |     - '        '
2026-03-14 09:56:26.735568 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.735572 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.735577 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.735581 | controller |     - '        '
2026-03-14 09:56:26.735585 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.735590 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.735606 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.735613 | controller |       available"'
2026-03-14 09:56:26.735618 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.735622 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.735626 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.735630 | controller |     - '          exit 1'
2026-03-14 09:56:26.735635 | controller |     - '        fi'
2026-03-14 09:56:26.735639 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.735643 | controller |     - '        '
2026-03-14 09:56:26.735648 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.735652 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.735656 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.735660 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.735665 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.735669 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.735673 | controller |       -- \'
2026-03-14 09:56:26.735678 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.735689 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.735693 | controller |     - '    logger.go:42: 09:43:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.735698 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.735702 | controller |     - '    logger.go:42: 09:43:57 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.735706 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.735711 | controller |       found'
2026-03-14 09:56:26.735715 | controller |     - '    logger.go:42: 09:43:58 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.735719 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.735723 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.735728 | controller |     - '        '
2026-03-14 09:56:26.735732 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.735736 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.735741 | controller |     - '        '
2026-03-14 09:56:26.735745 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.735751 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.735756 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.735760 | controller |     - '          exit 1'
2026-03-14 09:56:26.735764 | controller |     - '        fi'
2026-03-14 09:56:26.735768 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.735773 | controller |     - '        '
2026-03-14 09:56:26.735777 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.735781 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.735786 | controller |       | base64 -d)'
2026-03-14 09:56:26.735790 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.735794 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.735799 | controller |     - '          exit 1'
2026-03-14 09:56:26.735803 | controller |     - '        fi'
2026-03-14 09:56:26.735807 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.735812 | controller |     - '        '
2026-03-14 09:56:26.735816 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.735820 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.735824 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.735829 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.735833 | controller |       --timeout=300s'
2026-03-14 09:56:26.735837 | controller |     - '        '
2026-03-14 09:56:26.735842 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.735846 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.735850 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.735854 | controller |     - '        '
2026-03-14 09:56:26.735859 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.735863 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.735867 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.735872 | controller |     - '        '
2026-03-14 09:56:26.735876 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.735880 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.735885 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.735889 | controller |       available"'
2026-03-14 09:56:26.735893 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.735898 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.735904 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.735909 | controller |     - '          exit 1'
2026-03-14 09:56:26.735913 | controller |     - '        fi'
2026-03-14 09:56:26.735917 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.735922 | controller |     - '        '
2026-03-14 09:56:26.735926 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.735930 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.735934 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.735939 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.735945 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.735949 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.735953 | controller |       -- \'
2026-03-14 09:56:26.735958 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.735968 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.735973 | controller |     - '    logger.go:42: 09:43:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.735977 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.735981 | controller |     - '    logger.go:42: 09:43:58 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.735986 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.735993 | controller |       found'
2026-03-14 09:56:26.735997 | controller |     - '    logger.go:42: 09:43:59 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.736002 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.736006 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.736010 | controller |     - '        '
2026-03-14 09:56:26.736015 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.736019 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.736023 | controller |     - '        '
2026-03-14 09:56:26.736028 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.736032 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.736036 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.736040 | controller |     - '          exit 1'
2026-03-14 09:56:26.736045 | controller |     - '        fi'
2026-03-14 09:56:26.736049 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.736053 | controller |     - '        '
2026-03-14 09:56:26.736058 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.736062 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.736066 | controller |       | base64 -d)'
2026-03-14 09:56:26.736070 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.736075 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.736079 | controller |     - '          exit 1'
2026-03-14 09:56:26.736083 | controller |     - '        fi'
2026-03-14 09:56:26.736088 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.736092 | controller |     - '        '
2026-03-14 09:56:26.736097 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.736103 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.736109 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.736116 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.736122 | controller |       --timeout=300s'
2026-03-14 09:56:26.736127 | controller |     - '        '
2026-03-14 09:56:26.736131 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.736136 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.736142 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.736146 | controller |     - '        '
2026-03-14 09:56:26.736151 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.736157 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.736161 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.736165 | controller |     - '        '
2026-03-14 09:56:26.736170 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.736174 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.736178 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.736182 | controller |       available"'
2026-03-14 09:56:26.736187 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.736191 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.736195 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.736200 | controller |     - '          exit 1'
2026-03-14 09:56:26.736204 | controller |     - '        fi'
2026-03-14 09:56:26.736208 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.736213 | controller |     - '        '
2026-03-14 09:56:26.736217 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.736222 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.736226 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.736230 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.736235 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.736239 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.736243 | controller |       -- \'
2026-03-14 09:56:26.736248 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.736252 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.736264 | controller |     - '    logger.go:42: 09:43:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.736268 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.736273 | controller |     - '    logger.go:42: 09:43:59 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.736277 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.736282 | controller |       found'
2026-03-14 09:56:26.736286 | controller |     - '    logger.go:42: 09:44:00 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.736290 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.736295 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.736299 | controller |     - '        '
2026-03-14 09:56:26.736304 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.736308 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.736312 | controller |     - '        '
2026-03-14 09:56:26.736317 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.736321 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.736325 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.736330 | controller |     - '          exit 1'
2026-03-14 09:56:26.736334 | controller |     - '        fi'
2026-03-14 09:56:26.736338 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.736343 | controller |     - '        '
2026-03-14 09:56:26.736347 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.736353 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.736358 | controller |       | base64 -d)'
2026-03-14 09:56:26.736362 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.736367 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.736371 | controller |     - '          exit 1'
2026-03-14 09:56:26.736375 | controller |     - '        fi'
2026-03-14 09:56:26.736380 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.736385 | controller |     - '        '
2026-03-14 09:56:26.736389 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.736394 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.736398 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.736402 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.736407 | controller |       --timeout=300s'
2026-03-14 09:56:26.736411 | controller |     - '        '
2026-03-14 09:56:26.736416 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.736420 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.736424 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.736429 | controller |     - '        '
2026-03-14 09:56:26.736433 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.736438 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.736442 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.736448 | controller |     - '        '
2026-03-14 09:56:26.736453 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.736457 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.736462 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.736466 | controller |       available"'
2026-03-14 09:56:26.736470 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.736475 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.736479 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.736483 | controller |     - '          exit 1'
2026-03-14 09:56:26.736488 | controller |     - '        fi'
2026-03-14 09:56:26.736492 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.736496 | controller |     - '        '
2026-03-14 09:56:26.736501 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.736505 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.736509 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.736514 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.736518 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.736522 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.736527 | controller |       -- \'
2026-03-14 09:56:26.736531 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.736536 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.736548 | controller |     - '    logger.go:42: 09:44:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.736553 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.736557 | controller |     - '    logger.go:42: 09:44:01 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.736562 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.736566 | controller |       found'
2026-03-14 09:56:26.736570 | controller |     - '    logger.go:42: 09:44:02 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.736577 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.736581 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.736586 | controller |     - '        '
2026-03-14 09:56:26.736590 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.736605 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.736620 | controller |     - '        '
2026-03-14 09:56:26.736625 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.736630 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.736634 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.736639 | controller |     - '          exit 1'
2026-03-14 09:56:26.736643 | controller |     - '        fi'
2026-03-14 09:56:26.736647 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.736652 | controller |     - '        '
2026-03-14 09:56:26.736656 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.736661 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.736665 | controller |       | base64 -d)'
2026-03-14 09:56:26.736669 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.736674 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.736678 | controller |     - '          exit 1'
2026-03-14 09:56:26.736682 | controller |     - '        fi'
2026-03-14 09:56:26.736687 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.736691 | controller |     - '        '
2026-03-14 09:56:26.736695 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.736700 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.736704 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.736708 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.736713 | controller |       --timeout=300s'
2026-03-14 09:56:26.736717 | controller |     - '        '
2026-03-14 09:56:26.736722 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.736726 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.736730 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.736735 | controller |     - '        '
2026-03-14 09:56:26.736739 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.736744 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.736748 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.736752 | controller |     - '        '
2026-03-14 09:56:26.736757 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.736761 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.736768 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.736773 | controller |       available"'
2026-03-14 09:56:26.736777 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.736781 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.736786 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.736790 | controller |     - '          exit 1'
2026-03-14 09:56:26.736795 | controller |     - '        fi'
2026-03-14 09:56:26.736799 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.736803 | controller |     - '        '
2026-03-14 09:56:26.736808 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.736812 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.736816 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.736821 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.736825 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.736830 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.736837 | controller |       -- \'
2026-03-14 09:56:26.736842 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.736846 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.736851 | controller |     - '    logger.go:42: 09:44:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.736855 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.736867 | controller |     - '    logger.go:42: 09:44:02 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.736872 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.736876 | controller |       found'
2026-03-14 09:56:26.736881 | controller |     - '    logger.go:42: 09:44:03 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.736885 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.736889 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.736894 | controller |     - '        '
2026-03-14 09:56:26.736898 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.736903 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.736907 | controller |     - '        '
2026-03-14 09:56:26.736911 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.736916 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.736920 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.736924 | controller |     - '          exit 1'
2026-03-14 09:56:26.736929 | controller |     - '        fi'
2026-03-14 09:56:26.736935 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.736939 | controller |     - '        '
2026-03-14 09:56:26.736944 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.736948 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.736952 | controller |       | base64 -d)'
2026-03-14 09:56:26.736957 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.736961 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.736965 | controller |     - '          exit 1'
2026-03-14 09:56:26.736970 | controller |     - '        fi'
2026-03-14 09:56:26.736974 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.736978 | controller |     - '        '
2026-03-14 09:56:26.736984 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.736989 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.736993 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.736997 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.737002 | controller |       --timeout=300s'
2026-03-14 09:56:26.737006 | controller |     - '        '
2026-03-14 09:56:26.737011 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.737015 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.737019 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.737024 | controller |     - '        '
2026-03-14 09:56:26.737028 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.737033 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.737037 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.737041 | controller |     - '        '
2026-03-14 09:56:26.737046 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.737050 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.737054 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.737059 | controller |       available"'
2026-03-14 09:56:26.737063 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.737067 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.737072 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.737076 | controller |     - '          exit 1'
2026-03-14 09:56:26.737080 | controller |     - '        fi'
2026-03-14 09:56:26.737085 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.737089 | controller |     - '        '
2026-03-14 09:56:26.737094 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.737098 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.737102 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.737108 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.737114 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.737120 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.737127 | controller |       -- \'
2026-03-14 09:56:26.737133 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.737139 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.737144 | controller |     - '    logger.go:42: 09:44:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.737148 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.737160 | controller |     - '    logger.go:42: 09:44:03 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.737165 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.737169 | controller |       found'
2026-03-14 09:56:26.737174 | controller |     - '    logger.go:42: 09:44:04 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.737180 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.737184 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.737189 | controller |     - '        '
2026-03-14 09:56:26.737193 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.737200 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.737204 | controller |     - '        '
2026-03-14 09:56:26.737208 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.737213 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.737217 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.737221 | controller |     - '          exit 1'
2026-03-14 09:56:26.737226 | controller |     - '        fi'
2026-03-14 09:56:26.737230 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.737234 | controller |     - '        '
2026-03-14 09:56:26.737239 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.737245 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.737249 | controller |       | base64 -d)'
2026-03-14 09:56:26.737253 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.737258 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.737262 | controller |     - '          exit 1'
2026-03-14 09:56:26.737266 | controller |     - '        fi'
2026-03-14 09:56:26.737271 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.737275 | controller |     - '        '
2026-03-14 09:56:26.737279 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.737284 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.737288 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.737292 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.737297 | controller |       --timeout=300s'
2026-03-14 09:56:26.737301 | controller |     - '        '
2026-03-14 09:56:26.737305 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.737310 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.737314 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.737318 | controller |     - '        '
2026-03-14 09:56:26.737323 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.737327 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.737331 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.737336 | controller |     - '        '
2026-03-14 09:56:26.737340 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.737345 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.737349 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.737353 | controller |       available"'
2026-03-14 09:56:26.737357 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.737362 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.737366 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.737370 | controller |     - '          exit 1'
2026-03-14 09:56:26.737375 | controller |     - '        fi'
2026-03-14 09:56:26.737379 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.737385 | controller |     - '        '
2026-03-14 09:56:26.737389 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.737394 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.737398 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.737402 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.737407 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.737411 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.737415 | controller |       -- \'
2026-03-14 09:56:26.737420 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.737424 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.737429 | controller |     - '    logger.go:42: 09:44:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.737433 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.737437 | controller |     - '    logger.go:42: 09:44:04 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.737448 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.737453 | controller |       found'
2026-03-14 09:56:26.737457 | controller |     - '    logger.go:42: 09:44:05 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.737462 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.737466 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.737471 | controller |     - '        '
2026-03-14 09:56:26.737475 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.737479 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.737484 | controller |     - '        '
2026-03-14 09:56:26.737488 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.737492 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.737497 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.737501 | controller |     - '          exit 1'
2026-03-14 09:56:26.737505 | controller |     - '        fi'
2026-03-14 09:56:26.737510 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.737514 | controller |     - '        '
2026-03-14 09:56:26.737519 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.737523 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.737527 | controller |       | base64 -d)'
2026-03-14 09:56:26.737532 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.737536 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.737540 | controller |     - '          exit 1'
2026-03-14 09:56:26.737545 | controller |     - '        fi'
2026-03-14 09:56:26.737551 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.737555 | controller |     - '        '
2026-03-14 09:56:26.737560 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.737564 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.737568 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.737573 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.737577 | controller |       --timeout=300s'
2026-03-14 09:56:26.737581 | controller |     - '        '
2026-03-14 09:56:26.737586 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.737592 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.737609 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.737614 | controller |     - '        '
2026-03-14 09:56:26.737618 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.737623 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.737627 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.737631 | controller |     - '        '
2026-03-14 09:56:26.737636 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.737640 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.737644 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.737649 | controller |       available"'
2026-03-14 09:56:26.737653 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.737657 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.737662 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.737666 | controller |     - '          exit 1'
2026-03-14 09:56:26.737670 | controller |     - '        fi'
2026-03-14 09:56:26.737675 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.737679 | controller |     - '        '
2026-03-14 09:56:26.737683 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.737688 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.737692 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.737696 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.737701 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.737705 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.737709 | controller |       -- \'
2026-03-14 09:56:26.737714 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.737718 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.737722 | controller |     - '    logger.go:42: 09:44:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.737727 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.737731 | controller |     - '    logger.go:42: 09:44:05 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.737742 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.737746 | controller |       found'
2026-03-14 09:56:26.737751 | controller |     - '    logger.go:42: 09:44:06 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.737755 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.737760 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.737764 | controller |     - '        '
2026-03-14 09:56:26.737768 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.737773 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.737780 | controller |     - '        '
2026-03-14 09:56:26.737784 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.737790 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.737796 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.737800 | controller |     - '          exit 1'
2026-03-14 09:56:26.737805 | controller |     - '        fi'
2026-03-14 09:56:26.737809 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.737814 | controller |     - '        '
2026-03-14 09:56:26.737818 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.737822 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.737827 | controller |       | base64 -d)'
2026-03-14 09:56:26.737831 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.737836 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.737840 | controller |     - '          exit 1'
2026-03-14 09:56:26.737844 | controller |     - '        fi'
2026-03-14 09:56:26.737849 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.737853 | controller |     - '        '
2026-03-14 09:56:26.737858 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.737862 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.737866 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.737871 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.737875 | controller |       --timeout=300s'
2026-03-14 09:56:26.737879 | controller |     - '        '
2026-03-14 09:56:26.737884 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.737888 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.737892 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.737897 | controller |     - '        '
2026-03-14 09:56:26.737902 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.737908 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.737913 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.737918 | controller |     - '        '
2026-03-14 09:56:26.737922 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.737927 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.737931 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.737935 | controller |       available"'
2026-03-14 09:56:26.737940 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.737944 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.737949 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.737953 | controller |     - '          exit 1'
2026-03-14 09:56:26.737957 | controller |     - '        fi'
2026-03-14 09:56:26.737962 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.737967 | controller |     - '        '
2026-03-14 09:56:26.737972 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.737986 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.737997 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.738003 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.738013 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.738019 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.738025 | controller |       -- \'
2026-03-14 09:56:26.738031 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.738037 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.738043 | controller |     - '    logger.go:42: 09:44:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.738048 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.738055 | controller |     - '    logger.go:42: 09:44:06 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.738061 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.738067 | controller |       found'
2026-03-14 09:56:26.738083 | controller |     - '    logger.go:42: 09:44:07 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.738088 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.738093 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.738097 | controller |     - '        '
2026-03-14 09:56:26.738102 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.738108 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.738113 | controller |     - '        '
2026-03-14 09:56:26.738118 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.738122 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.738126 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.738132 | controller |     - '          exit 1'
2026-03-14 09:56:26.738138 | controller |     - '        fi'
2026-03-14 09:56:26.738145 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.738151 | controller |     - '        '
2026-03-14 09:56:26.738157 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.738162 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.738166 | controller |       | base64 -d)'
2026-03-14 09:56:26.738171 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.738175 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.738179 | controller |     - '          exit 1'
2026-03-14 09:56:26.738183 | controller |     - '        fi'
2026-03-14 09:56:26.738188 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.738192 | controller |     - '        '
2026-03-14 09:56:26.738196 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.738201 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.738206 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.738212 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.738216 | controller |       --timeout=300s'
2026-03-14 09:56:26.738220 | controller |     - '        '
2026-03-14 09:56:26.738225 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.738233 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.738237 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.738244 | controller |     - '        '
2026-03-14 09:56:26.738248 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.738253 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.738259 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.738263 | controller |     - '        '
2026-03-14 09:56:26.738268 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.738272 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.738276 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.738280 | controller |       available"'
2026-03-14 09:56:26.738285 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.738289 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.738293 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.738297 | controller |     - '          exit 1'
2026-03-14 09:56:26.738302 | controller |     - '        fi'
2026-03-14 09:56:26.738306 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.738310 | controller |     - '        '
2026-03-14 09:56:26.738314 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.738319 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.738323 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.738327 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.738331 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.738336 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.738341 | controller |       -- \'
2026-03-14 09:56:26.738348 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.738352 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.738357 | controller |     - '    logger.go:42: 09:44:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.738362 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.738367 | controller |     - '    logger.go:42: 09:44:08 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.738372 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.738376 | controller |       found'
2026-03-14 09:56:26.738388 | controller |     - '    logger.go:42: 09:44:09 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.738393 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.738397 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.738402 | controller |     - '        '
2026-03-14 09:56:26.738406 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.738410 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.738414 | controller |     - '        '
2026-03-14 09:56:26.738419 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.738425 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.738429 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.738433 | controller |     - '          exit 1'
2026-03-14 09:56:26.738438 | controller |     - '        fi'
2026-03-14 09:56:26.738442 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.738446 | controller |     - '        '
2026-03-14 09:56:26.738450 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.738455 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.738461 | controller |       | base64 -d)'
2026-03-14 09:56:26.738465 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.738469 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.738474 | controller |     - '          exit 1'
2026-03-14 09:56:26.738478 | controller |     - '        fi'
2026-03-14 09:56:26.738482 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.738486 | controller |     - '        '
2026-03-14 09:56:26.738491 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.738495 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.738499 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.738503 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.738508 | controller |       --timeout=300s'
2026-03-14 09:56:26.738512 | controller |     - '        '
2026-03-14 09:56:26.738516 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.738520 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.738525 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.738529 | controller |     - '        '
2026-03-14 09:56:26.738533 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.738537 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.738541 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.738546 | controller |     - '        '
2026-03-14 09:56:26.738550 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.738554 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.738558 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.738563 | controller |       available"'
2026-03-14 09:56:26.738567 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.738571 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.738577 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.738581 | controller |     - '          exit 1'
2026-03-14 09:56:26.738585 | controller |     - '        fi'
2026-03-14 09:56:26.738590 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.738607 | controller |     - '        '
2026-03-14 09:56:26.738615 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.738619 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.738624 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.738628 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.738632 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.738636 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.738641 | controller |       -- \'
2026-03-14 09:56:26.738645 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.738649 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.738656 | controller |     - '    logger.go:42: 09:44:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.738660 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.738664 | controller |     - '    logger.go:42: 09:44:09 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.738668 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.738673 | controller |       found'
2026-03-14 09:56:26.738677 | controller |     - '    logger.go:42: 09:44:10 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.738681 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.738692 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.738697 | controller |     - '        '
2026-03-14 09:56:26.738702 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.738706 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.738710 | controller |     - '        '
2026-03-14 09:56:26.738715 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.738719 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.738723 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.738728 | controller |     - '          exit 1'
2026-03-14 09:56:26.738739 | controller |     - '        fi'
2026-03-14 09:56:26.738743 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.738747 | controller |     - '        '
2026-03-14 09:56:26.738752 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.738756 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.738760 | controller |       | base64 -d)'
2026-03-14 09:56:26.738765 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.738769 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.738773 | controller |     - '          exit 1'
2026-03-14 09:56:26.738777 | controller |     - '        fi'
2026-03-14 09:56:26.738781 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.738786 | controller |     - '        '
2026-03-14 09:56:26.738790 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.738794 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.738798 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.738802 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.738807 | controller |       --timeout=300s'
2026-03-14 09:56:26.738811 | controller |     - '        '
2026-03-14 09:56:26.738815 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.738819 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.738824 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.738828 | controller |     - '        '
2026-03-14 09:56:26.738832 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.738836 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.738841 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.738845 | controller |     - '        '
2026-03-14 09:56:26.738849 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.738853 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.738857 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.738864 | controller |       available"'
2026-03-14 09:56:26.738868 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.738872 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.738876 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.738881 | controller |     - '          exit 1'
2026-03-14 09:56:26.738885 | controller |     - '        fi'
2026-03-14 09:56:26.738889 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.738893 | controller |     - '        '
2026-03-14 09:56:26.738898 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.738902 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.738906 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.738910 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.738914 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.738919 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.738923 | controller |       -- \'
2026-03-14 09:56:26.738927 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.738931 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.738936 | controller |     - '    logger.go:42: 09:44:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.738940 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.738944 | controller |     - '    logger.go:42: 09:44:10 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.738948 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.738953 | controller |       found'
2026-03-14 09:56:26.738957 | controller |     - '    logger.go:42: 09:44:11 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.738961 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.738965 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.738970 | controller |     - '        '
2026-03-14 09:56:26.738980 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.738985 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.738989 | controller |     - '        '
2026-03-14 09:56:26.738993 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.738998 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.739013 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.739019 | controller |     - '          exit 1'
2026-03-14 09:56:26.739025 | controller |     - '        fi'
2026-03-14 09:56:26.739029 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.739033 | controller |     - '        '
2026-03-14 09:56:26.739038 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.739042 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.739046 | controller |       | base64 -d)'
2026-03-14 09:56:26.739050 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.739055 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.739059 | controller |     - '          exit 1'
2026-03-14 09:56:26.739063 | controller |     - '        fi'
2026-03-14 09:56:26.739067 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.739072 | controller |     - '        '
2026-03-14 09:56:26.739076 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.739082 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.739089 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.739093 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.739097 | controller |       --timeout=300s'
2026-03-14 09:56:26.739101 | controller |     - '        '
2026-03-14 09:56:26.739106 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.739110 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.739114 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.739118 | controller |     - '        '
2026-03-14 09:56:26.739123 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.739129 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.739134 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.739138 | controller |     - '        '
2026-03-14 09:56:26.739142 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.739148 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.739154 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.739160 | controller |       available"'
2026-03-14 09:56:26.739166 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.739172 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.739177 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.739182 | controller |     - '          exit 1'
2026-03-14 09:56:26.739186 | controller |     - '        fi'
2026-03-14 09:56:26.739190 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.739194 | controller |     - '        '
2026-03-14 09:56:26.739198 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.739203 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.739207 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.739211 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.739215 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.739222 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.739227 | controller |       -- \'
2026-03-14 09:56:26.739231 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.739235 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.739239 | controller |     - '    logger.go:42: 09:44:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.739244 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.739248 | controller |     - '    logger.go:42: 09:44:11 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.739252 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.739256 | controller |       found'
2026-03-14 09:56:26.739261 | controller |     - '    logger.go:42: 09:44:12 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.739265 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.739271 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.739275 | controller |     - '        '
2026-03-14 09:56:26.739288 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.739292 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.739297 | controller |     - '        '
2026-03-14 09:56:26.739301 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.739305 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.739310 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.739314 | controller |     - '          exit 1'
2026-03-14 09:56:26.739318 | controller |     - '        fi'
2026-03-14 09:56:26.739322 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.739327 | controller |     - '        '
2026-03-14 09:56:26.739331 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.739335 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.739339 | controller |       | base64 -d)'
2026-03-14 09:56:26.739344 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.739348 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.739352 | controller |     - '          exit 1'
2026-03-14 09:56:26.739356 | controller |     - '        fi'
2026-03-14 09:56:26.739361 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.739365 | controller |     - '        '
2026-03-14 09:56:26.739369 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.739374 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.739378 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.739382 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.739386 | controller |       --timeout=300s'
2026-03-14 09:56:26.739392 | controller |     - '        '
2026-03-14 09:56:26.739397 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.739401 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.739405 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.739410 | controller |     - '        '
2026-03-14 09:56:26.739414 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.739418 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.739422 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.739427 | controller |     - '        '
2026-03-14 09:56:26.739431 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.739435 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.739439 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.739444 | controller |       available"'
2026-03-14 09:56:26.739448 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.739452 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.739456 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.739461 | controller |     - '          exit 1'
2026-03-14 09:56:26.739465 | controller |     - '        fi'
2026-03-14 09:56:26.739469 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.739475 | controller |     - '        '
2026-03-14 09:56:26.739479 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.739484 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.739488 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.739492 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.739496 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.739501 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.739505 | controller |       -- \'
2026-03-14 09:56:26.739509 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.739513 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.739518 | controller |     - '    logger.go:42: 09:44:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.739522 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.739526 | controller |     - '    logger.go:42: 09:44:12 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.739530 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.739535 | controller |       found'
2026-03-14 09:56:26.739539 | controller |     - '    logger.go:42: 09:44:13 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.739543 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.739547 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.739551 | controller |     - '        '
2026-03-14 09:56:26.739556 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.739566 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.739571 | controller |     - '        '
2026-03-14 09:56:26.739575 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.739579 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.739584 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.739588 | controller |     - '          exit 1'
2026-03-14 09:56:26.739592 | controller |     - '        fi'
2026-03-14 09:56:26.739610 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.739618 | controller |     - '        '
2026-03-14 09:56:26.739622 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.739626 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.739633 | controller |       | base64 -d)'
2026-03-14 09:56:26.739638 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.739642 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.739646 | controller |     - '          exit 1'
2026-03-14 09:56:26.739650 | controller |     - '        fi'
2026-03-14 09:56:26.739655 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.739659 | controller |     - '        '
2026-03-14 09:56:26.739663 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.739667 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.739672 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.739677 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.739682 | controller |       --timeout=300s'
2026-03-14 09:56:26.739686 | controller |     - '        '
2026-03-14 09:56:26.739690 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.739696 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.739700 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.739705 | controller |     - '        '
2026-03-14 09:56:26.739709 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.739713 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.739717 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.739722 | controller |     - '        '
2026-03-14 09:56:26.739726 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.739730 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.739734 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.739739 | controller |       available"'
2026-03-14 09:56:26.739743 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.739747 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.739752 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.739756 | controller |     - '          exit 1'
2026-03-14 09:56:26.739762 | controller |     - '        fi'
2026-03-14 09:56:26.739766 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.739770 | controller |     - '        '
2026-03-14 09:56:26.739775 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.739779 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.739783 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.739787 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.739791 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.739796 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.739800 | controller |       -- \'
2026-03-14 09:56:26.739804 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.739808 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.739813 | controller |     - '    logger.go:42: 09:44:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.739817 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.739821 | controller |     - '    logger.go:42: 09:44:13 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.739825 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.739830 | controller |       found'
2026-03-14 09:56:26.739834 | controller |     - '    logger.go:42: 09:44:14 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.739838 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.739842 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.739847 | controller |     - '        '
2026-03-14 09:56:26.739851 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.739862 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.739867 | controller |     - '        '
2026-03-14 09:56:26.739871 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.739876 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.739883 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.739889 | controller |     - '          exit 1'
2026-03-14 09:56:26.739895 | controller |     - '        fi'
2026-03-14 09:56:26.739901 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.739907 | controller |     - '        '
2026-03-14 09:56:26.739913 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.739922 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.739928 | controller |       | base64 -d)'
2026-03-14 09:56:26.739934 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.739940 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.739946 | controller |     - '          exit 1'
2026-03-14 09:56:26.739950 | controller |     - '        fi'
2026-03-14 09:56:26.739954 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.739959 | controller |     - '        '
2026-03-14 09:56:26.739963 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.739967 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.739971 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.739975 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.739980 | controller |       --timeout=300s'
2026-03-14 09:56:26.739984 | controller |     - '        '
2026-03-14 09:56:26.739988 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.739992 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.739997 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.740001 | controller |     - '        '
2026-03-14 09:56:26.740005 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.740009 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.740014 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.740018 | controller |     - '        '
2026-03-14 09:56:26.740022 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.740026 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.740031 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.740035 | controller |       available"'
2026-03-14 09:56:26.740039 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.740044 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.740048 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.740052 | controller |     - '          exit 1'
2026-03-14 09:56:26.740056 | controller |     - '        fi'
2026-03-14 09:56:26.740061 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.740065 | controller |     - '        '
2026-03-14 09:56:26.740069 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.740073 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.740078 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.740082 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.740086 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.740093 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.740097 | controller |       -- \'
2026-03-14 09:56:26.740101 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.740106 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.740110 | controller |     - '    logger.go:42: 09:44:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.740114 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.740118 | controller |     - '    logger.go:42: 09:44:15 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.740123 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.740127 | controller |       found'
2026-03-14 09:56:26.740131 | controller |     - '    logger.go:42: 09:44:16 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.740135 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.740140 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.740144 | controller |     - '        '
2026-03-14 09:56:26.740150 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.740156 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.740160 | controller |     - '        '
2026-03-14 09:56:26.740175 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.740183 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.740189 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.740195 | controller |     - '          exit 1'
2026-03-14 09:56:26.740199 | controller |     - '        fi'
2026-03-14 09:56:26.740203 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.740207 | controller |     - '        '
2026-03-14 09:56:26.740212 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.740216 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.740220 | controller |       | base64 -d)'
2026-03-14 09:56:26.740224 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.740229 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.740233 | controller |     - '          exit 1'
2026-03-14 09:56:26.740237 | controller |     - '        fi'
2026-03-14 09:56:26.740241 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.740246 | controller |     - '        '
2026-03-14 09:56:26.740250 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.740254 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.740258 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.740263 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.740270 | controller |       --timeout=300s'
2026-03-14 09:56:26.740274 | controller |     - '        '
2026-03-14 09:56:26.740278 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.740282 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.740287 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.740291 | controller |     - '        '
2026-03-14 09:56:26.740295 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.740300 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.740307 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.740311 | controller |     - '        '
2026-03-14 09:56:26.740316 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.740320 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.740324 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.740329 | controller |       available"'
2026-03-14 09:56:26.740333 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.740337 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.740341 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.740346 | controller |     - '          exit 1'
2026-03-14 09:56:26.740350 | controller |     - '        fi'
2026-03-14 09:56:26.740354 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.740358 | controller |     - '        '
2026-03-14 09:56:26.740363 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.740367 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.740371 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.740375 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.740380 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.740384 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.740388 | controller |       -- \'
2026-03-14 09:56:26.740392 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.740397 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.740401 | controller |     - '    logger.go:42: 09:44:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.740405 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.740409 | controller |     - '    logger.go:42: 09:44:16 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.740414 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.740418 | controller |       found'
2026-03-14 09:56:26.740422 | controller |     - '    logger.go:42: 09:44:17 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.740426 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.740431 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.740435 | controller |     - '        '
2026-03-14 09:56:26.740439 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.740443 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.740448 | controller |     - '        '
2026-03-14 09:56:26.740459 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.740464 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.740468 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.740473 | controller |     - '          exit 1'
2026-03-14 09:56:26.740477 | controller |     - '        fi'
2026-03-14 09:56:26.740481 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.740485 | controller |     - '        '
2026-03-14 09:56:26.740490 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.740494 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.740498 | controller |       | base64 -d)'
2026-03-14 09:56:26.740504 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.740509 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.740513 | controller |     - '          exit 1'
2026-03-14 09:56:26.740517 | controller |     - '        fi'
2026-03-14 09:56:26.740521 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.740526 | controller |     - '        '
2026-03-14 09:56:26.740530 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.740534 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.740538 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.740543 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.740547 | controller |       --timeout=300s'
2026-03-14 09:56:26.740551 | controller |     - '        '
2026-03-14 09:56:26.740555 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.740560 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.740564 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.740568 | controller |     - '        '
2026-03-14 09:56:26.740572 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.740577 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.740581 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.740585 | controller |     - '        '
2026-03-14 09:56:26.740590 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.740606 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.740613 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.740617 | controller |       available"'
2026-03-14 09:56:26.740621 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.740626 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.740630 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.740634 | controller |     - '          exit 1'
2026-03-14 09:56:26.740638 | controller |     - '        fi'
2026-03-14 09:56:26.740643 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.740647 | controller |     - '        '
2026-03-14 09:56:26.740651 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.740655 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.740660 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.740664 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.740668 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.740672 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.740677 | controller |       -- \'
2026-03-14 09:56:26.740681 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.740685 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.740690 | controller |     - '    logger.go:42: 09:44:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.740696 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.740700 | controller |     - '    logger.go:42: 09:44:17 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.740704 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.740709 | controller |       found'
2026-03-14 09:56:26.740713 | controller |     - '    logger.go:42: 09:44:18 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.740717 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.740721 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.740725 | controller |     - '        '
2026-03-14 09:56:26.740730 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.740734 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.740738 | controller |     - '        '
2026-03-14 09:56:26.740742 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.740747 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.740758 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.740762 | controller |     - '          exit 1'
2026-03-14 09:56:26.740767 | controller |     - '        fi'
2026-03-14 09:56:26.740771 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.740775 | controller |     - '        '
2026-03-14 09:56:26.740779 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.740784 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.740788 | controller |       | base64 -d)'
2026-03-14 09:56:26.740792 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.740796 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.740801 | controller |     - '          exit 1'
2026-03-14 09:56:26.740805 | controller |     - '        fi'
2026-03-14 09:56:26.740809 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.740814 | controller |     - '        '
2026-03-14 09:56:26.740818 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.740822 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.740826 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.740831 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.740835 | controller |       --timeout=300s'
2026-03-14 09:56:26.740841 | controller |     - '        '
2026-03-14 09:56:26.740847 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.740852 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.740858 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.740862 | controller |     - '        '
2026-03-14 09:56:26.740866 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.740870 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.740875 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.740879 | controller |     - '        '
2026-03-14 09:56:26.740883 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.740889 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.740893 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.740898 | controller |       available"'
2026-03-14 09:56:26.740904 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.740909 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.740913 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.740917 | controller |     - '          exit 1'
2026-03-14 09:56:26.740922 | controller |     - '        fi'
2026-03-14 09:56:26.740926 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.740930 | controller |     - '        '
2026-03-14 09:56:26.740934 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.740939 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.740943 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.740947 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.740951 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.740956 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.740963 | controller |       -- \'
2026-03-14 09:56:26.740967 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.740971 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.740976 | controller |     - '    logger.go:42: 09:44:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.740980 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.740984 | controller |     - '    logger.go:42: 09:44:18 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.740988 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.740993 | controller |       found'
2026-03-14 09:56:26.740997 | controller |     - '    logger.go:42: 09:44:19 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.741001 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.741005 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.741010 | controller |     - '        '
2026-03-14 09:56:26.741014 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.741018 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.741022 | controller |     - '        '
2026-03-14 09:56:26.741027 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.741031 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.741042 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.741046 | controller |     - '          exit 1'
2026-03-14 09:56:26.741050 | controller |     - '        fi'
2026-03-14 09:56:26.741055 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.741059 | controller |     - '        '
2026-03-14 09:56:26.741063 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.741067 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.741072 | controller |       | base64 -d)'
2026-03-14 09:56:26.741076 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.741080 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.741084 | controller |     - '          exit 1'
2026-03-14 09:56:26.741089 | controller |     - '        fi'
2026-03-14 09:56:26.741093 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.741097 | controller |     - '        '
2026-03-14 09:56:26.741101 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.741106 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.741112 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.741116 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.741120 | controller |       --timeout=300s'
2026-03-14 09:56:26.741125 | controller |     - '        '
2026-03-14 09:56:26.741129 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.741133 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.741137 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.741142 | controller |     - '        '
2026-03-14 09:56:26.741146 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.741150 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.741154 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.741159 | controller |     - '        '
2026-03-14 09:56:26.741163 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.741167 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.741171 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.741176 | controller |       available"'
2026-03-14 09:56:26.741181 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.741187 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.741195 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.741202 | controller |     - '          exit 1'
2026-03-14 09:56:26.741208 | controller |     - '        fi'
2026-03-14 09:56:26.741212 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.741216 | controller |     - '        '
2026-03-14 09:56:26.741221 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.741225 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.741229 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.741233 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.741238 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.741242 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.741246 | controller |       -- \'
2026-03-14 09:56:26.741250 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.741257 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.741261 | controller |     - '    logger.go:42: 09:44:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.741265 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.741269 | controller |     - '    logger.go:42: 09:44:19 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.741274 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.741278 | controller |       found'
2026-03-14 09:56:26.741282 | controller |     - '    logger.go:42: 09:44:20 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.741286 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.741290 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.741296 | controller |     - '        '
2026-03-14 09:56:26.741301 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.741305 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.741309 | controller |     - '        '
2026-03-14 09:56:26.741313 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.741318 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.741322 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.741326 | controller |     - '          exit 1'
2026-03-14 09:56:26.741338 | controller |     - '        fi'
2026-03-14 09:56:26.741343 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.741347 | controller |     - '        '
2026-03-14 09:56:26.741351 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.741356 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.741362 | controller |       | base64 -d)'
2026-03-14 09:56:26.741367 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.741371 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.741376 | controller |     - '          exit 1'
2026-03-14 09:56:26.741382 | controller |     - '        fi'
2026-03-14 09:56:26.741386 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.741390 | controller |     - '        '
2026-03-14 09:56:26.741395 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.741399 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.741403 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.741407 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.741411 | controller |       --timeout=300s'
2026-03-14 09:56:26.741416 | controller |     - '        '
2026-03-14 09:56:26.741420 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.741424 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.741428 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.741433 | controller |     - '        '
2026-03-14 09:56:26.741437 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.741441 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.741445 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.741450 | controller |     - '        '
2026-03-14 09:56:26.741454 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.741458 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.741462 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.741490 | controller |       available"'
2026-03-14 09:56:26.741495 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.741499 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.741503 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.741507 | controller |     - '          exit 1'
2026-03-14 09:56:26.741512 | controller |     - '        fi'
2026-03-14 09:56:26.741516 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.741522 | controller |     - '        '
2026-03-14 09:56:26.741527 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.741534 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.741538 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.741542 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.741546 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.741551 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.741555 | controller |       -- \'
2026-03-14 09:56:26.741559 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.741563 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.741568 | controller |     - '    logger.go:42: 09:44:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.741572 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.741576 | controller |     - '    logger.go:42: 09:44:20 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.741581 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.741585 | controller |       found'
2026-03-14 09:56:26.741589 | controller |     - '    logger.go:42: 09:44:21 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.741605 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.741613 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.741618 | controller |     - '        '
2026-03-14 09:56:26.741622 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.741627 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.741632 | controller |     - '        '
2026-03-14 09:56:26.741638 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.741643 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.741647 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.741651 | controller |     - '          exit 1'
2026-03-14 09:56:26.741656 | controller |     - '        fi'
2026-03-14 09:56:26.741668 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.741672 | controller |     - '        '
2026-03-14 09:56:26.741677 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.741681 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.741685 | controller |       | base64 -d)'
2026-03-14 09:56:26.741690 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.741694 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.741698 | controller |     - '          exit 1'
2026-03-14 09:56:26.741702 | controller |     - '        fi'
2026-03-14 09:56:26.741707 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.741711 | controller |     - '        '
2026-03-14 09:56:26.741715 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.741719 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.741724 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.741728 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.741732 | controller |       --timeout=300s'
2026-03-14 09:56:26.741736 | controller |     - '        '
2026-03-14 09:56:26.741741 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.741745 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.741751 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.741755 | controller |     - '        '
2026-03-14 09:56:26.741760 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.741764 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.741768 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.741772 | controller |     - '        '
2026-03-14 09:56:26.741777 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.741781 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.741785 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.741789 | controller |       available"'
2026-03-14 09:56:26.741794 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.741798 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.741802 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.741808 | controller |     - '          exit 1'
2026-03-14 09:56:26.741813 | controller |     - '        fi'
2026-03-14 09:56:26.741817 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.741821 | controller |     - '        '
2026-03-14 09:56:26.741825 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.741830 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.741834 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.741838 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.741842 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.741847 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.741851 | controller |       -- \'
2026-03-14 09:56:26.741855 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.741859 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.741864 | controller |     - '    logger.go:42: 09:44:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.741868 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.741872 | controller |     - '    logger.go:42: 09:44:22 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.741876 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.741880 | controller |       found'
2026-03-14 09:56:26.741885 | controller |     - '    logger.go:42: 09:44:23 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.741889 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.741893 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.741898 | controller |     - '        '
2026-03-14 09:56:26.741903 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.741909 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.741913 | controller |     - '        '
2026-03-14 09:56:26.741917 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.741921 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.741926 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.741932 | controller |     - '          exit 1'
2026-03-14 09:56:26.741936 | controller |     - '        fi'
2026-03-14 09:56:26.741940 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.741945 | controller |     - '        '
2026-03-14 09:56:26.741957 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.741962 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.741966 | controller |       | base64 -d)'
2026-03-14 09:56:26.741970 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.741975 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.741979 | controller |     - '          exit 1'
2026-03-14 09:56:26.741983 | controller |     - '        fi'
2026-03-14 09:56:26.741987 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.741992 | controller |     - '        '
2026-03-14 09:56:26.741996 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.742000 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.742004 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.742009 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.742013 | controller |       --timeout=300s'
2026-03-14 09:56:26.742017 | controller |     - '        '
2026-03-14 09:56:26.742021 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.742025 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.742030 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.742034 | controller |     - '        '
2026-03-14 09:56:26.742038 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.742042 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.742047 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.742051 | controller |     - '        '
2026-03-14 09:56:26.742057 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.742062 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.742067 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.742071 | controller |       available"'
2026-03-14 09:56:26.742075 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.742079 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.742084 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.742088 | controller |     - '          exit 1'
2026-03-14 09:56:26.742092 | controller |     - '        fi'
2026-03-14 09:56:26.742096 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.742103 | controller |     - '        '
2026-03-14 09:56:26.742107 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.742111 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.742115 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.742119 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.742124 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.742130 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.742134 | controller |       -- \'
2026-03-14 09:56:26.742138 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.742143 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.742147 | controller |     - '    logger.go:42: 09:44:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.742151 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.742155 | controller |     - '    logger.go:42: 09:44:23 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.742160 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.742164 | controller |       found'
2026-03-14 09:56:26.742168 | controller |     - '    logger.go:42: 09:44:24 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.742172 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.742176 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.742181 | controller |     - '        '
2026-03-14 09:56:26.742185 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.742189 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.742193 | controller |     - '        '
2026-03-14 09:56:26.742199 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.742205 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.742211 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.742218 | controller |     - '          exit 1'
2026-03-14 09:56:26.742224 | controller |     - '        fi'
2026-03-14 09:56:26.742228 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.742233 | controller |     - '        '
2026-03-14 09:56:26.742237 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.742248 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.742253 | controller |       | base64 -d)'
2026-03-14 09:56:26.742257 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.742262 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.742266 | controller |     - '          exit 1'
2026-03-14 09:56:26.742270 | controller |     - '        fi'
2026-03-14 09:56:26.742274 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.742279 | controller |     - '        '
2026-03-14 09:56:26.742283 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.742287 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.742291 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.742296 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.742300 | controller |       --timeout=300s'
2026-03-14 09:56:26.742304 | controller |     - '        '
2026-03-14 09:56:26.742309 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.742315 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.742319 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.742324 | controller |     - '        '
2026-03-14 09:56:26.742328 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.742332 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.742336 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.742346 | controller |     - '        '
2026-03-14 09:56:26.742350 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.742355 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.742359 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.742363 | controller |       available"'
2026-03-14 09:56:26.742367 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.742371 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.742376 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.742380 | controller |     - '          exit 1'
2026-03-14 09:56:26.742384 | controller |     - '        fi'
2026-03-14 09:56:26.742388 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.742393 | controller |     - '        '
2026-03-14 09:56:26.742397 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.742401 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.742405 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.742410 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.742414 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.742418 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.742422 | controller |       -- \'
2026-03-14 09:56:26.742427 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.742431 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.742435 | controller |     - '    logger.go:42: 09:44:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.742439 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.742444 | controller |     - '    logger.go:42: 09:44:24 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.742448 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.742452 | controller |       found'
2026-03-14 09:56:26.742456 | controller |     - '    logger.go:42: 09:44:25 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.742460 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.742465 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.742469 | controller |     - '        '
2026-03-14 09:56:26.742473 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.742477 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.742482 | controller |     - '        '
2026-03-14 09:56:26.742486 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.742490 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.742494 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.742499 | controller |     - '          exit 1'
2026-03-14 09:56:26.742503 | controller |     - '        fi'
2026-03-14 09:56:26.742507 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.742511 | controller |     - '        '
2026-03-14 09:56:26.742516 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.742528 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.742532 | controller |       | base64 -d)'
2026-03-14 09:56:26.742538 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.742546 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.742553 | controller |     - '          exit 1'
2026-03-14 09:56:26.742559 | controller |     - '        fi'
2026-03-14 09:56:26.742565 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.742571 | controller |     - '        '
2026-03-14 09:56:26.742577 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.742583 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.742588 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.742592 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.742611 | controller |       --timeout=300s'
2026-03-14 09:56:26.742615 | controller |     - '        '
2026-03-14 09:56:26.742620 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.742624 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.742628 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.742632 | controller |     - '        '
2026-03-14 09:56:26.742637 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.742641 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.742645 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.742649 | controller |     - '        '
2026-03-14 09:56:26.742654 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.742658 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.742662 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.742666 | controller |       available"'
2026-03-14 09:56:26.742671 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.742677 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.742683 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.742689 | controller |     - '          exit 1'
2026-03-14 09:56:26.742695 | controller |     - '        fi'
2026-03-14 09:56:26.742701 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.742707 | controller |     - '        '
2026-03-14 09:56:26.742713 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.742722 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.742726 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.742731 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.742735 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.742739 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.742743 | controller |       -- \'
2026-03-14 09:56:26.742748 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.742752 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.742756 | controller |     - '    logger.go:42: 09:44:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.742760 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.742767 | controller |     - '    logger.go:42: 09:44:25 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.742771 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.742775 | controller |       found'
2026-03-14 09:56:26.742780 | controller |     - '    logger.go:42: 09:44:26 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.742784 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.742788 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.742792 | controller |     - '        '
2026-03-14 09:56:26.742797 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.742801 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.742805 | controller |     - '        '
2026-03-14 09:56:26.742809 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.742813 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.742818 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.742822 | controller |     - '          exit 1'
2026-03-14 09:56:26.742826 | controller |     - '        fi'
2026-03-14 09:56:26.742830 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.742835 | controller |     - '        '
2026-03-14 09:56:26.742839 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.742843 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.742856 | controller |       | base64 -d)'
2026-03-14 09:56:26.742861 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.742865 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.742870 | controller |     - '          exit 1'
2026-03-14 09:56:26.742874 | controller |     - '        fi'
2026-03-14 09:56:26.742878 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.742882 | controller |     - '        '
2026-03-14 09:56:26.742887 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.742891 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.742895 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.742899 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.742903 | controller |       --timeout=300s'
2026-03-14 09:56:26.742909 | controller |     - '        '
2026-03-14 09:56:26.742915 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.742921 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.742927 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.742933 | controller |     - '        '
2026-03-14 09:56:26.742939 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.742945 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.742951 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.742957 | controller |     - '        '
2026-03-14 09:56:26.742961 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.742965 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.742969 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.742974 | controller |       available"'
2026-03-14 09:56:26.742978 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.742985 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.742989 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.742993 | controller |     - '          exit 1'
2026-03-14 09:56:26.742997 | controller |     - '        fi'
2026-03-14 09:56:26.743002 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.743019 | controller |     - '        '
2026-03-14 09:56:26.743027 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.743031 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.743036 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.743040 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.743044 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.743048 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.743053 | controller |       -- \'
2026-03-14 09:56:26.743057 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.743061 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.743065 | controller |     - '    logger.go:42: 09:44:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.743070 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.743074 | controller |     - '    logger.go:42: 09:44:26 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.743078 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.743082 | controller |       found'
2026-03-14 09:56:26.743087 | controller |     - '    logger.go:42: 09:44:27 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.743091 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.743095 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.743099 | controller |     - '        '
2026-03-14 09:56:26.743103 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.743108 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.743112 | controller |     - '        '
2026-03-14 09:56:26.743116 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.743120 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.743125 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.743129 | controller |     - '          exit 1'
2026-03-14 09:56:26.743133 | controller |     - '        fi'
2026-03-14 09:56:26.743137 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.743142 | controller |     - '        '
2026-03-14 09:56:26.743146 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.743150 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.743154 | controller |       | base64 -d)'
2026-03-14 09:56:26.743166 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.743171 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.743175 | controller |     - '          exit 1'
2026-03-14 09:56:26.743180 | controller |     - '        fi'
2026-03-14 09:56:26.743184 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.743188 | controller |     - '        '
2026-03-14 09:56:26.743192 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.743196 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.743203 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.743207 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.743211 | controller |       --timeout=300s'
2026-03-14 09:56:26.743217 | controller |     - '        '
2026-03-14 09:56:26.743223 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.743229 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.743236 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.743242 | controller |     - '        '
2026-03-14 09:56:26.743247 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.743251 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.743256 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.743260 | controller |     - '        '
2026-03-14 09:56:26.743264 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.743268 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.743273 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.743277 | controller |       available"'
2026-03-14 09:56:26.743281 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.743285 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.743290 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.743294 | controller |     - '          exit 1'
2026-03-14 09:56:26.743298 | controller |     - '        fi'
2026-03-14 09:56:26.743302 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.743306 | controller |     - '        '
2026-03-14 09:56:26.743311 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.743315 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.743319 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.743323 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.743330 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.743334 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.743339 | controller |       -- \'
2026-03-14 09:56:26.743343 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.743347 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.743351 | controller |     - '    logger.go:42: 09:44:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.743356 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.743360 | controller |     - '    logger.go:42: 09:44:27 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.743364 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.743368 | controller |       found'
2026-03-14 09:56:26.743373 | controller |     - '    logger.go:42: 09:44:28 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.743377 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.743381 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.743385 | controller |     - '        '
2026-03-14 09:56:26.743392 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.743396 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.743400 | controller |     - '        '
2026-03-14 09:56:26.743404 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.743409 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.743413 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.743417 | controller |     - '          exit 1'
2026-03-14 09:56:26.743421 | controller |     - '        fi'
2026-03-14 09:56:26.743426 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.743430 | controller |     - '        '
2026-03-14 09:56:26.743434 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.743438 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.743443 | controller |       | base64 -d)'
2026-03-14 09:56:26.743447 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.743458 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.743463 | controller |     - '          exit 1'
2026-03-14 09:56:26.743467 | controller |     - '        fi'
2026-03-14 09:56:26.743472 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.743476 | controller |     - '        '
2026-03-14 09:56:26.743480 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.743484 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.743489 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.743493 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.743497 | controller |       --timeout=300s'
2026-03-14 09:56:26.743501 | controller |     - '        '
2026-03-14 09:56:26.743506 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.743510 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.743524 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.743528 | controller |     - '        '
2026-03-14 09:56:26.743532 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.743536 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.743541 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.743545 | controller |     - '        '
2026-03-14 09:56:26.743549 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.743553 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.743558 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.743562 | controller |       available"'
2026-03-14 09:56:26.743566 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.743570 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.743575 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.743579 | controller |     - '          exit 1'
2026-03-14 09:56:26.743583 | controller |     - '        fi'
2026-03-14 09:56:26.743587 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.743592 | controller |     - '        '
2026-03-14 09:56:26.743610 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.743616 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.743621 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.743625 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.743629 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.743634 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.743638 | controller |       -- \'
2026-03-14 09:56:26.743642 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.743646 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.743651 | controller |     - '    logger.go:42: 09:44:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.743655 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.743659 | controller |     - '    logger.go:42: 09:44:28 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.743663 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.743668 | controller |       found'
2026-03-14 09:56:26.743672 | controller |     - '    logger.go:42: 09:44:29 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.743676 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.743680 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.743685 | controller |     - '        '
2026-03-14 09:56:26.743689 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.743693 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.743697 | controller |     - '        '
2026-03-14 09:56:26.743702 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.743706 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.743710 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.743715 | controller |     - '          exit 1'
2026-03-14 09:56:26.743719 | controller |     - '        fi'
2026-03-14 09:56:26.743723 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.743729 | controller |     - '        '
2026-03-14 09:56:26.743733 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.743738 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.743743 | controller |       | base64 -d)'
2026-03-14 09:56:26.743748 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.743759 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.743764 | controller |     - '          exit 1'
2026-03-14 09:56:26.743768 | controller |     - '        fi'
2026-03-14 09:56:26.743773 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.743777 | controller |     - '        '
2026-03-14 09:56:26.743781 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.743785 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.743790 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.743794 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.743798 | controller |       --timeout=300s'
2026-03-14 09:56:26.743802 | controller |     - '        '
2026-03-14 09:56:26.743807 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.743811 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.743818 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.743822 | controller |     - '        '
2026-03-14 09:56:26.743827 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.743831 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.743835 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.743839 | controller |     - '        '
2026-03-14 09:56:26.743844 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.743848 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.743852 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.743856 | controller |       available"'
2026-03-14 09:56:26.743861 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.743865 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.743871 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.743875 | controller |     - '          exit 1'
2026-03-14 09:56:26.743879 | controller |     - '        fi'
2026-03-14 09:56:26.743883 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.743888 | controller |     - '        '
2026-03-14 09:56:26.743892 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.743896 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.743900 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.743905 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.743909 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.743913 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.743917 | controller |       -- \'
2026-03-14 09:56:26.743921 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.743926 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.743933 | controller |     - '    logger.go:42: 09:44:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.743937 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.743941 | controller |     - '    logger.go:42: 09:44:30 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.743946 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.743950 | controller |       found'
2026-03-14 09:56:26.743954 | controller |     - '    logger.go:42: 09:44:31 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.743958 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.743963 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.743967 | controller |     - '        '
2026-03-14 09:56:26.743971 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.743975 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.743980 | controller |     - '        '
2026-03-14 09:56:26.743984 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.743988 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.743994 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.744000 | controller |     - '          exit 1'
2026-03-14 09:56:26.744009 | controller |     - '        fi'
2026-03-14 09:56:26.744015 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.744021 | controller |     - '        '
2026-03-14 09:56:26.744027 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.744033 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.744039 | controller |       | base64 -d)'
2026-03-14 09:56:26.744045 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.744052 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.744058 | controller |     - '          exit 1'
2026-03-14 09:56:26.744062 | controller |     - '        fi'
2026-03-14 09:56:26.744076 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.744083 | controller |     - '        '
2026-03-14 09:56:26.744089 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.744095 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.744100 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.744106 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.744112 | controller |       --timeout=300s'
2026-03-14 09:56:26.744118 | controller |     - '        '
2026-03-14 09:56:26.744124 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.744130 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.744141 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.744146 | controller |     - '        '
2026-03-14 09:56:26.744150 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.744155 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.744159 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.744163 | controller |     - '        '
2026-03-14 09:56:26.744167 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.744172 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.744176 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.744180 | controller |       available"'
2026-03-14 09:56:26.744184 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.744189 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.744193 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.744197 | controller |     - '          exit 1'
2026-03-14 09:56:26.744201 | controller |     - '        fi'
2026-03-14 09:56:26.744206 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.744210 | controller |     - '        '
2026-03-14 09:56:26.744214 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.744218 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.744223 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.744227 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.744232 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.744238 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.744248 | controller |       -- \'
2026-03-14 09:56:26.744254 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.744261 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.744265 | controller |     - '    logger.go:42: 09:44:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.744269 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.744274 | controller |     - '    logger.go:42: 09:44:31 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.744280 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.744284 | controller |       found'
2026-03-14 09:56:26.744288 | controller |     - '    logger.go:42: 09:44:32 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.744293 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.744297 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.744301 | controller |     - '        '
2026-03-14 09:56:26.744305 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.744310 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.744314 | controller |     - '        '
2026-03-14 09:56:26.744318 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.744322 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.744327 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.744331 | controller |     - '          exit 1'
2026-03-14 09:56:26.744335 | controller |     - '        fi'
2026-03-14 09:56:26.744341 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.744346 | controller |     - '        '
2026-03-14 09:56:26.744350 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.744355 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.744360 | controller |       | base64 -d)'
2026-03-14 09:56:26.744365 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.744370 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.744374 | controller |     - '          exit 1'
2026-03-14 09:56:26.744378 | controller |     - '        fi'
2026-03-14 09:56:26.744391 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.744396 | controller |     - '        '
2026-03-14 09:56:26.744400 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.744405 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.744409 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.744413 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.744417 | controller |       --timeout=300s'
2026-03-14 09:56:26.744421 | controller |     - '        '
2026-03-14 09:56:26.744426 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.744430 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.744434 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.744438 | controller |     - '        '
2026-03-14 09:56:26.744443 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.744447 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.744451 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.744458 | controller |     - '        '
2026-03-14 09:56:26.744462 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.744467 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.744471 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.744475 | controller |       available"'
2026-03-14 09:56:26.744479 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.744484 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.744488 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.744492 | controller |     - '          exit 1'
2026-03-14 09:56:26.744496 | controller |     - '        fi'
2026-03-14 09:56:26.744500 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.744505 | controller |     - '        '
2026-03-14 09:56:26.744509 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.744513 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.744517 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.744524 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.744528 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.744532 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.744537 | controller |       -- \'
2026-03-14 09:56:26.744541 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.744545 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.744549 | controller |     - '    logger.go:42: 09:44:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.744554 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.744558 | controller |     - '    logger.go:42: 09:44:32 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.744564 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.744568 | controller |       found'
2026-03-14 09:56:26.744573 | controller |     - '    logger.go:42: 09:44:33 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.744577 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.744581 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.744585 | controller |     - '        '
2026-03-14 09:56:26.744589 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.744606 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.744612 | controller |     - '        '
2026-03-14 09:56:26.744617 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.744621 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.744625 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.744630 | controller |     - '          exit 1'
2026-03-14 09:56:26.744634 | controller |     - '        fi'
2026-03-14 09:56:26.744638 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.744642 | controller |     - '        '
2026-03-14 09:56:26.744647 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.744651 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.744655 | controller |       | base64 -d)'
2026-03-14 09:56:26.744659 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.744664 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.744670 | controller |     - '          exit 1'
2026-03-14 09:56:26.744674 | controller |     - '        fi'
2026-03-14 09:56:26.744679 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.744683 | controller |     - '        '
2026-03-14 09:56:26.744694 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.744699 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.744703 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.744707 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.744711 | controller |       --timeout=300s'
2026-03-14 09:56:26.744716 | controller |     - '        '
2026-03-14 09:56:26.744720 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.744724 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.744728 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.744733 | controller |     - '        '
2026-03-14 09:56:26.744737 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.744741 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.744745 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.744750 | controller |     - '        '
2026-03-14 09:56:26.744754 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.744758 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.744762 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.744767 | controller |       available"'
2026-03-14 09:56:26.744771 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.744775 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.744779 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.744783 | controller |     - '          exit 1'
2026-03-14 09:56:26.744788 | controller |     - '        fi'
2026-03-14 09:56:26.744792 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.744796 | controller |     - '        '
2026-03-14 09:56:26.744800 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.744805 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.744809 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.744813 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.744817 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.744822 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.744826 | controller |       -- \'
2026-03-14 09:56:26.744830 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.744835 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.744839 | controller |     - '    logger.go:42: 09:44:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.744843 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.744847 | controller |     - '    logger.go:42: 09:44:33 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.744853 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.744858 | controller |       found'
2026-03-14 09:56:26.744862 | controller |     - '    logger.go:42: 09:44:34 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.744866 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.744870 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.744875 | controller |     - '        '
2026-03-14 09:56:26.744879 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.744883 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.744887 | controller |     - '        '
2026-03-14 09:56:26.744891 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.744896 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.744900 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.744904 | controller |     - '          exit 1'
2026-03-14 09:56:26.744908 | controller |     - '        fi'
2026-03-14 09:56:26.744913 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.744917 | controller |     - '        '
2026-03-14 09:56:26.744921 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.744925 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.744929 | controller |       | base64 -d)'
2026-03-14 09:56:26.744934 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.744938 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.744942 | controller |     - '          exit 1'
2026-03-14 09:56:26.744946 | controller |     - '        fi'
2026-03-14 09:56:26.744951 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.744955 | controller |     - '        '
2026-03-14 09:56:26.744959 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.744972 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.744976 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.744981 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.744985 | controller |       --timeout=300s'
2026-03-14 09:56:26.744989 | controller |     - '        '
2026-03-14 09:56:26.744993 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.744997 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.745002 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.745006 | controller |     - '        '
2026-03-14 09:56:26.745010 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.745014 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.745019 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.745023 | controller |     - '        '
2026-03-14 09:56:26.745027 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.745032 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.745036 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.745040 | controller |       available"'
2026-03-14 09:56:26.745044 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.745049 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.745055 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.745059 | controller |     - '          exit 1'
2026-03-14 09:56:26.745064 | controller |     - '        fi'
2026-03-14 09:56:26.745068 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.745072 | controller |     - '        '
2026-03-14 09:56:26.745076 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.745080 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.745085 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.745089 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.745093 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.745097 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.745104 | controller |       -- \'
2026-03-14 09:56:26.745108 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.745113 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.745117 | controller |     - '    logger.go:42: 09:44:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.745121 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.745125 | controller |     - '    logger.go:42: 09:44:34 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.745129 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.745136 | controller |       found'
2026-03-14 09:56:26.745140 | controller |     - '    logger.go:42: 09:44:35 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.745144 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.745148 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.745153 | controller |     - '        '
2026-03-14 09:56:26.745157 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.745161 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.745165 | controller |     - '        '
2026-03-14 09:56:26.745170 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.745174 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.745181 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.745186 | controller |     - '          exit 1'
2026-03-14 09:56:26.745190 | controller |     - '        fi'
2026-03-14 09:56:26.745194 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.745199 | controller |     - '        '
2026-03-14 09:56:26.745203 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.745209 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.745213 | controller |       | base64 -d)'
2026-03-14 09:56:26.745218 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.745222 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.745226 | controller |     - '          exit 1'
2026-03-14 09:56:26.745232 | controller |     - '        fi'
2026-03-14 09:56:26.745237 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.745241 | controller |     - '        '
2026-03-14 09:56:26.745245 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.745259 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.745266 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.745275 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.745280 | controller |       --timeout=300s'
2026-03-14 09:56:26.745285 | controller |     - '        '
2026-03-14 09:56:26.745289 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.745293 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.745298 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.745302 | controller |     - '        '
2026-03-14 09:56:26.745306 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.745311 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.745315 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.745319 | controller |     - '        '
2026-03-14 09:56:26.745323 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.745328 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.745332 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.745336 | controller |       available"'
2026-03-14 09:56:26.745340 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.745345 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.745349 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.745353 | controller |     - '          exit 1'
2026-03-14 09:56:26.745358 | controller |     - '        fi'
2026-03-14 09:56:26.745362 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.745366 | controller |     - '        '
2026-03-14 09:56:26.745370 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.745375 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.745379 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.745383 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.745388 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.745392 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.745396 | controller |       -- \'
2026-03-14 09:56:26.745400 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.745405 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.745409 | controller |     - '    logger.go:42: 09:44:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.745413 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.745418 | controller |     - '    logger.go:42: 09:44:35 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.745422 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.745427 | controller |       found'
2026-03-14 09:56:26.745431 | controller |     - '    logger.go:42: 09:44:36 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.745435 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.745440 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.745444 | controller |     - '        '
2026-03-14 09:56:26.745449 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.745454 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.745459 | controller |     - '        '
2026-03-14 09:56:26.745463 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.745467 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.745472 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.745476 | controller |     - '          exit 1'
2026-03-14 09:56:26.745480 | controller |     - '        fi'
2026-03-14 09:56:26.745484 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.745489 | controller |     - '        '
2026-03-14 09:56:26.745493 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.745497 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.745502 | controller |       | base64 -d)'
2026-03-14 09:56:26.745506 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.745510 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.745514 | controller |     - '          exit 1'
2026-03-14 09:56:26.745519 | controller |     - '        fi'
2026-03-14 09:56:26.745523 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.745527 | controller |     - '        '
2026-03-14 09:56:26.745531 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.745536 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.745547 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.745551 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.745563 | controller |       --timeout=300s'
2026-03-14 09:56:26.745567 | controller |     - '        '
2026-03-14 09:56:26.745572 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.745576 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.745581 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.745587 | controller |     - '        '
2026-03-14 09:56:26.745591 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.745610 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.745614 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.745618 | controller |     - '        '
2026-03-14 09:56:26.745623 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.745627 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.745631 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.745635 | controller |       available"'
2026-03-14 09:56:26.745639 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.745644 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.745648 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.745652 | controller |     - '          exit 1'
2026-03-14 09:56:26.745656 | controller |     - '        fi'
2026-03-14 09:56:26.745661 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.745665 | controller |     - '        '
2026-03-14 09:56:26.745669 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.745673 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.745680 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.745684 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.745688 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.745692 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.745697 | controller |       -- \'
2026-03-14 09:56:26.745701 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.745705 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.745709 | controller |     - '    logger.go:42: 09:44:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.745714 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.745718 | controller |     - '    logger.go:42: 09:44:37 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.745722 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.745726 | controller |       found'
2026-03-14 09:56:26.745731 | controller |     - '    logger.go:42: 09:44:38 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.745737 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.745741 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.745746 | controller |     - '        '
2026-03-14 09:56:26.745750 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.745754 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.745758 | controller |     - '        '
2026-03-14 09:56:26.745763 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.745767 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.745771 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.745775 | controller |     - '          exit 1'
2026-03-14 09:56:26.745779 | controller |     - '        fi'
2026-03-14 09:56:26.745784 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.745788 | controller |     - '        '
2026-03-14 09:56:26.745792 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.745796 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.745801 | controller |       | base64 -d)'
2026-03-14 09:56:26.745805 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.745809 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.745813 | controller |     - '          exit 1'
2026-03-14 09:56:26.745818 | controller |     - '        fi'
2026-03-14 09:56:26.745822 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.745826 | controller |     - '        '
2026-03-14 09:56:26.745830 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.745835 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.745845 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.745850 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.745854 | controller |       --timeout=300s'
2026-03-14 09:56:26.745859 | controller |     - '        '
2026-03-14 09:56:26.745863 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.745867 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.745872 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.745878 | controller |     - '        '
2026-03-14 09:56:26.745882 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.745886 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.745890 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.745895 | controller |     - '        '
2026-03-14 09:56:26.745899 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.745903 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.745907 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.745912 | controller |       available"'
2026-03-14 09:56:26.745916 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.745920 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.745924 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.745929 | controller |     - '          exit 1'
2026-03-14 09:56:26.745933 | controller |     - '        fi'
2026-03-14 09:56:26.745937 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.745941 | controller |     - '        '
2026-03-14 09:56:26.745945 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.745950 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.745954 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.745958 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.745962 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.745967 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.745971 | controller |       -- \'
2026-03-14 09:56:26.745975 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.745979 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.745984 | controller |     - '    logger.go:42: 09:44:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.745988 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.745992 | controller |     - '    logger.go:42: 09:44:38 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.745996 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.746001 | controller |       found'
2026-03-14 09:56:26.746005 | controller |     - '    logger.go:42: 09:44:39 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.746009 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.746013 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.746018 | controller |     - '        '
2026-03-14 09:56:26.746022 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.746026 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.746030 | controller |     - '        '
2026-03-14 09:56:26.746034 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.746039 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.746043 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.746047 | controller |     - '          exit 1'
2026-03-14 09:56:26.746060 | controller |     - '        fi'
2026-03-14 09:56:26.746066 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.746070 | controller |     - '        '
2026-03-14 09:56:26.746074 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.746079 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.746115 | controller |       | base64 -d)'
2026-03-14 09:56:26.746121 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.746127 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.746132 | controller |     - '          exit 1'
2026-03-14 09:56:26.746136 | controller |     - '        fi'
2026-03-14 09:56:26.746140 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.746145 | controller |     - '        '
2026-03-14 09:56:26.746149 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.746153 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.746158 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.746169 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.746174 | controller |       --timeout=300s'
2026-03-14 09:56:26.746178 | controller |     - '        '
2026-03-14 09:56:26.746182 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.746188 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.746193 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.746198 | controller |     - '        '
2026-03-14 09:56:26.746206 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.746221 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.746226 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.746230 | controller |     - '        '
2026-03-14 09:56:26.746236 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.746240 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.746245 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.746250 | controller |       available"'
2026-03-14 09:56:26.746255 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.746259 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.746264 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.746270 | controller |     - '          exit 1'
2026-03-14 09:56:26.746277 | controller |     - '        fi'
2026-03-14 09:56:26.746283 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.746290 | controller |     - '        '
2026-03-14 09:56:26.746295 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.746300 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.746305 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.746310 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.746314 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.746319 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.746330 | controller |       -- \'
2026-03-14 09:56:26.746363 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.746372 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.746376 | controller |     - '    logger.go:42: 09:44:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.746382 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.746386 | controller |     - '    logger.go:42: 09:44:39 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.746391 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.746395 | controller |       found'
2026-03-14 09:56:26.746399 | controller |     - '    logger.go:42: 09:44:40 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.746404 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.746408 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.746413 | controller |     - '        '
2026-03-14 09:56:26.746418 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.746425 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.746430 | controller |     - '        '
2026-03-14 09:56:26.746435 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.746439 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.746444 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.746448 | controller |     - '          exit 1'
2026-03-14 09:56:26.746452 | controller |     - '        fi'
2026-03-14 09:56:26.746456 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.746461 | controller |     - '        '
2026-03-14 09:56:26.746465 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.746469 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.746473 | controller |       | base64 -d)'
2026-03-14 09:56:26.746478 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.746482 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.746486 | controller |     - '          exit 1'
2026-03-14 09:56:26.746490 | controller |     - '        fi'
2026-03-14 09:56:26.746495 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.746499 | controller |     - '        '
2026-03-14 09:56:26.746503 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.746507 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.746512 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.746516 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.746528 | controller |       --timeout=300s'
2026-03-14 09:56:26.746532 | controller |     - '        '
2026-03-14 09:56:26.746537 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.746543 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.746547 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.746551 | controller |     - '        '
2026-03-14 09:56:26.746555 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.746560 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.746564 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.746568 | controller |     - '        '
2026-03-14 09:56:26.746575 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.746579 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.746583 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.746587 | controller |       available"'
2026-03-14 09:56:26.746592 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.746612 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.746620 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.746630 | controller |     - '          exit 1'
2026-03-14 09:56:26.746634 | controller |     - '        fi'
2026-03-14 09:56:26.746639 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.746647 | controller |     - '        '
2026-03-14 09:56:26.746651 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.746655 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.746659 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.746664 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.746668 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.746674 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.746679 | controller |       -- \'
2026-03-14 09:56:26.746683 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.746687 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.746691 | controller |     - '    logger.go:42: 09:44:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.746696 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.746700 | controller |     - '    logger.go:42: 09:44:40 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.746704 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.746708 | controller |       found'
2026-03-14 09:56:26.746713 | controller |     - '    logger.go:42: 09:44:41 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.746717 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.746721 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.746726 | controller |     - '        '
2026-03-14 09:56:26.746730 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.746734 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.746738 | controller |     - '        '
2026-03-14 09:56:26.746742 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.746747 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.746751 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.746755 | controller |     - '          exit 1'
2026-03-14 09:56:26.746759 | controller |     - '        fi'
2026-03-14 09:56:26.746764 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.746768 | controller |     - '        '
2026-03-14 09:56:26.746774 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.746778 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.746782 | controller |       | base64 -d)'
2026-03-14 09:56:26.746787 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.746791 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.746797 | controller |     - '          exit 1'
2026-03-14 09:56:26.746802 | controller |     - '        fi'
2026-03-14 09:56:26.746806 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.746810 | controller |     - '        '
2026-03-14 09:56:26.746815 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.746819 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.746823 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.746827 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.746831 | controller |       --timeout=300s'
2026-03-14 09:56:26.746836 | controller |     - '        '
2026-03-14 09:56:26.746848 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.746853 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.746857 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.746862 | controller |     - '        '
2026-03-14 09:56:26.746866 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.746870 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.746874 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.746879 | controller |     - '        '
2026-03-14 09:56:26.746883 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.746887 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.746892 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.746896 | controller |       available"'
2026-03-14 09:56:26.746900 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.746904 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.746909 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.746913 | controller |     - '          exit 1'
2026-03-14 09:56:26.746917 | controller |     - '        fi'
2026-03-14 09:56:26.746921 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.746926 | controller |     - '        '
2026-03-14 09:56:26.746930 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.746934 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.746939 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.746943 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.746947 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.746951 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.746956 | controller |       -- \'
2026-03-14 09:56:26.746960 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.746964 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.746969 | controller |     - '    logger.go:42: 09:44:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.746973 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.746977 | controller |     - '    logger.go:42: 09:44:41 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.746983 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.746987 | controller |       found'
2026-03-14 09:56:26.746992 | controller |     - '    logger.go:42: 09:44:42 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.746996 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.747000 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.747004 | controller |     - '        '
2026-03-14 09:56:26.747076 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.747082 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.747090 | controller |     - '        '
2026-03-14 09:56:26.747094 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.747099 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.747103 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.747107 | controller |     - '          exit 1'
2026-03-14 09:56:26.747112 | controller |     - '        fi'
2026-03-14 09:56:26.747116 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.747120 | controller |     - '        '
2026-03-14 09:56:26.747124 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.747128 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.747133 | controller |       | base64 -d)'
2026-03-14 09:56:26.747137 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.747147 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.747151 | controller |     - '          exit 1'
2026-03-14 09:56:26.747155 | controller |     - '        fi'
2026-03-14 09:56:26.747163 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.747168 | controller |     - '        '
2026-03-14 09:56:26.747172 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.747176 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.747181 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.747185 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.747189 | controller |       --timeout=300s'
2026-03-14 09:56:26.747193 | controller |     - '        '
2026-03-14 09:56:26.747205 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.747210 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.747214 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.747218 | controller |     - '        '
2026-03-14 09:56:26.747223 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.747227 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.747231 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.747236 | controller |     - '        '
2026-03-14 09:56:26.747240 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.747244 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.747248 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.747252 | controller |       available"'
2026-03-14 09:56:26.747257 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.747261 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.747267 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.747272 | controller |     - '          exit 1'
2026-03-14 09:56:26.747276 | controller |     - '        fi'
2026-03-14 09:56:26.747281 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.747287 | controller |     - '        '
2026-03-14 09:56:26.747293 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.747299 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.747306 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.747311 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.747316 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.747320 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.747324 | controller |       -- \'
2026-03-14 09:56:26.747329 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.747333 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.747337 | controller |     - '    logger.go:42: 09:44:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.747341 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.747346 | controller |     - '    logger.go:42: 09:44:42 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.747350 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.747354 | controller |       found'
2026-03-14 09:56:26.747358 | controller |     - '    logger.go:42: 09:44:44 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.747363 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.747367 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.747371 | controller |     - '        '
2026-03-14 09:56:26.747375 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.747380 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.747384 | controller |     - '        '
2026-03-14 09:56:26.747388 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.747392 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.747397 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.747401 | controller |     - '          exit 1'
2026-03-14 09:56:26.747405 | controller |     - '        fi'
2026-03-14 09:56:26.747409 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.747414 | controller |     - '        '
2026-03-14 09:56:26.747418 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.747422 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.747426 | controller |       | base64 -d)'
2026-03-14 09:56:26.747431 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.747435 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.747439 | controller |     - '          exit 1'
2026-03-14 09:56:26.747443 | controller |     - '        fi'
2026-03-14 09:56:26.747448 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.747452 | controller |     - '        '
2026-03-14 09:56:26.747456 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.747460 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.747465 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.747471 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.747475 | controller |       --timeout=300s'
2026-03-14 09:56:26.747479 | controller |     - '        '
2026-03-14 09:56:26.747484 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.747495 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.747500 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.747504 | controller |     - '        '
2026-03-14 09:56:26.747508 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.747513 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.747517 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.747521 | controller |     - '        '
2026-03-14 09:56:26.747526 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.747530 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.747534 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.747539 | controller |       available"'
2026-03-14 09:56:26.747545 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.747550 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.747554 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.747558 | controller |     - '          exit 1'
2026-03-14 09:56:26.747562 | controller |     - '        fi'
2026-03-14 09:56:26.747566 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.747571 | controller |     - '        '
2026-03-14 09:56:26.747575 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.747579 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.747583 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.747588 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.747592 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.747624 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.747631 | controller |       -- \'
2026-03-14 09:56:26.747642 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.747649 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.747653 | controller |     - '    logger.go:42: 09:44:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.747658 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.747662 | controller |     - '    logger.go:42: 09:44:44 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.747666 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.747670 | controller |       found'
2026-03-14 09:56:26.747675 | controller |     - '    logger.go:42: 09:44:45 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.747679 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.747683 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.747687 | controller |     - '        '
2026-03-14 09:56:26.747692 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.747696 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.747703 | controller |     - '        '
2026-03-14 09:56:26.747707 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.747713 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.747718 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.747722 | controller |     - '          exit 1'
2026-03-14 09:56:26.747726 | controller |     - '        fi'
2026-03-14 09:56:26.747730 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.747735 | controller |     - '        '
2026-03-14 09:56:26.747739 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.747743 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.747747 | controller |       | base64 -d)'
2026-03-14 09:56:26.747752 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.747756 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.747760 | controller |     - '          exit 1'
2026-03-14 09:56:26.747764 | controller |     - '        fi'
2026-03-14 09:56:26.747769 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.747773 | controller |     - '        '
2026-03-14 09:56:26.747777 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.747781 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.747786 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.747790 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.747794 | controller |       --timeout=300s'
2026-03-14 09:56:26.747798 | controller |     - '        '
2026-03-14 09:56:26.747803 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.747816 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.747821 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.747826 | controller |     - '        '
2026-03-14 09:56:26.747830 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.747834 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.747840 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.747845 | controller |     - '        '
2026-03-14 09:56:26.747849 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.747853 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.747858 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.747862 | controller |       available"'
2026-03-14 09:56:26.747867 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.747873 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.747879 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.747885 | controller |     - '          exit 1'
2026-03-14 09:56:26.747891 | controller |     - '        fi'
2026-03-14 09:56:26.747897 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.747903 | controller |     - '        '
2026-03-14 09:56:26.747910 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.747915 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.747919 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.747926 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.747930 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.747934 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.747939 | controller |       -- \'
2026-03-14 09:56:26.747943 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.747947 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.747952 | controller |     - '    logger.go:42: 09:44:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.747956 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.747960 | controller |     - '    logger.go:42: 09:44:46 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.747964 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.747969 | controller |       found'
2026-03-14 09:56:26.747973 | controller |     - '    logger.go:42: 09:44:47 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.747977 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.747982 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.747986 | controller |     - '        '
2026-03-14 09:56:26.747990 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.747994 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.747999 | controller |     - '        '
2026-03-14 09:56:26.748003 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.748007 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.748011 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.748016 | controller |     - '          exit 1'
2026-03-14 09:56:26.748020 | controller |     - '        fi'
2026-03-14 09:56:26.748024 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.748031 | controller |     - '        '
2026-03-14 09:56:26.748035 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.748041 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.748048 | controller |       | base64 -d)'
2026-03-14 09:56:26.748053 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.748057 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.748061 | controller |     - '          exit 1'
2026-03-14 09:56:26.748065 | controller |     - '        fi'
2026-03-14 09:56:26.748070 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.748074 | controller |     - '        '
2026-03-14 09:56:26.748078 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.748082 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.748087 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.748091 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.748095 | controller |       --timeout=300s'
2026-03-14 09:56:26.748099 | controller |     - '        '
2026-03-14 09:56:26.748104 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.748108 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.748121 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.748128 | controller |     - '        '
2026-03-14 09:56:26.748132 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.748136 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.748141 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.748145 | controller |     - '        '
2026-03-14 09:56:26.748149 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.748153 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.748158 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.748162 | controller |       available"'
2026-03-14 09:56:26.748166 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.748171 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.748175 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.748179 | controller |     - '          exit 1'
2026-03-14 09:56:26.748183 | controller |     - '        fi'
2026-03-14 09:56:26.748188 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.748192 | controller |     - '        '
2026-03-14 09:56:26.748196 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.748200 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.748204 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.748209 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.748213 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.748217 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.748222 | controller |       -- \'
2026-03-14 09:56:26.748226 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.748230 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.748234 | controller |     - '    logger.go:42: 09:44:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.748239 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.748243 | controller |     - '    logger.go:42: 09:44:47 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.748247 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.748251 | controller |       found'
2026-03-14 09:56:26.748256 | controller |     - '    logger.go:42: 09:44:48 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.748260 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.748264 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.748268 | controller |     - '        '
2026-03-14 09:56:26.748273 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.748277 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.748281 | controller |     - '        '
2026-03-14 09:56:26.748286 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.748290 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.748294 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.748300 | controller |     - '          exit 1'
2026-03-14 09:56:26.748306 | controller |     - '        fi'
2026-03-14 09:56:26.748312 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.748321 | controller |     - '        '
2026-03-14 09:56:26.748327 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.748331 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.748336 | controller |       | base64 -d)'
2026-03-14 09:56:26.748340 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.748344 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.748349 | controller |     - '          exit 1'
2026-03-14 09:56:26.748353 | controller |     - '        fi'
2026-03-14 09:56:26.748357 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.748362 | controller |     - '        '
2026-03-14 09:56:26.748366 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.748370 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.748377 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.748381 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.748385 | controller |       --timeout=300s'
2026-03-14 09:56:26.748389 | controller |     - '        '
2026-03-14 09:56:26.748394 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.748398 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.748410 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.748414 | controller |     - '        '
2026-03-14 09:56:26.748419 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.748423 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.748427 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.748432 | controller |     - '        '
2026-03-14 09:56:26.748436 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.748440 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.748444 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.748448 | controller |       available"'
2026-03-14 09:56:26.748453 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.748457 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.748461 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.748465 | controller |     - '          exit 1'
2026-03-14 09:56:26.748469 | controller |     - '        fi'
2026-03-14 09:56:26.748474 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.748478 | controller |     - '        '
2026-03-14 09:56:26.748482 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.748486 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.748491 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.748495 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.748499 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.748503 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.748508 | controller |       -- \'
2026-03-14 09:56:26.748512 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.748519 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.748523 | controller |     - '    logger.go:42: 09:44:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.748528 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.748532 | controller |     - '    logger.go:42: 09:44:48 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.748536 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.748542 | controller |       found'
2026-03-14 09:56:26.748548 | controller |     - '    logger.go:42: 09:44:49 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.748554 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.748560 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.748566 | controller |     - '        '
2026-03-14 09:56:26.748572 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.748578 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.748584 | controller |     - '        '
2026-03-14 09:56:26.748590 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.748607 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.748614 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.748618 | controller |     - '          exit 1'
2026-03-14 09:56:26.748623 | controller |     - '        fi'
2026-03-14 09:56:26.748627 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.748631 | controller |     - '        '
2026-03-14 09:56:26.748635 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.748640 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.748644 | controller |       | base64 -d)'
2026-03-14 09:56:26.748648 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.748652 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.748657 | controller |     - '          exit 1'
2026-03-14 09:56:26.748661 | controller |     - '        fi'
2026-03-14 09:56:26.748665 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.748670 | controller |     - '        '
2026-03-14 09:56:26.748677 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.748682 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.748686 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.748690 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.748694 | controller |       --timeout=300s'
2026-03-14 09:56:26.748699 | controller |     - '        '
2026-03-14 09:56:26.748703 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.748707 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.748721 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.748725 | controller |     - '        '
2026-03-14 09:56:26.748730 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.748734 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.748738 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.748742 | controller |     - '        '
2026-03-14 09:56:26.748747 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.748753 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.748758 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.748762 | controller |       available"'
2026-03-14 09:56:26.748766 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.748770 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.748775 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.748779 | controller |     - '          exit 1'
2026-03-14 09:56:26.748783 | controller |     - '        fi'
2026-03-14 09:56:26.748787 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.748791 | controller |     - '        '
2026-03-14 09:56:26.748796 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.748800 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.748804 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.748808 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.748813 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.748817 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.748821 | controller |       -- \'
2026-03-14 09:56:26.748826 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.748832 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.748836 | controller |     - '    logger.go:42: 09:44:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.748840 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.748845 | controller |     - '    logger.go:42: 09:44:49 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.748849 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.748853 | controller |       found'
2026-03-14 09:56:26.748857 | controller |     - '    logger.go:42: 09:44:50 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.748862 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.748866 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.748870 | controller |     - '        '
2026-03-14 09:56:26.748874 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.748879 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.748883 | controller |     - '        '
2026-03-14 09:56:26.748887 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.748891 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.748895 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.748900 | controller |     - '          exit 1'
2026-03-14 09:56:26.748904 | controller |     - '        fi'
2026-03-14 09:56:26.748908 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.748913 | controller |     - '        '
2026-03-14 09:56:26.748917 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.748921 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.748925 | controller |       | base64 -d)'
2026-03-14 09:56:26.748930 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.748934 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.748938 | controller |     - '          exit 1'
2026-03-14 09:56:26.748944 | controller |     - '        fi'
2026-03-14 09:56:26.748948 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.748952 | controller |     - '        '
2026-03-14 09:56:26.748957 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.748961 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.748965 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.748971 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.748975 | controller |       --timeout=300s'
2026-03-14 09:56:26.748980 | controller |     - '        '
2026-03-14 09:56:26.748984 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.748988 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.748993 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.748997 | controller |     - '        '
2026-03-14 09:56:26.749007 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.749012 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.749016 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.749021 | controller |     - '        '
2026-03-14 09:56:26.749025 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.749029 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.749034 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.749038 | controller |       available"'
2026-03-14 09:56:26.749042 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.749046 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.749051 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.749055 | controller |     - '          exit 1'
2026-03-14 09:56:26.749059 | controller |     - '        fi'
2026-03-14 09:56:26.749063 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.749068 | controller |     - '        '
2026-03-14 09:56:26.749072 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.749076 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.749080 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.749084 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.749089 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.749093 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.749097 | controller |       -- \'
2026-03-14 09:56:26.749101 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.749106 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.749110 | controller |     - '    logger.go:42: 09:44:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.749114 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.749119 | controller |     - '    logger.go:42: 09:44:50 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.749123 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.749129 | controller |       found'
2026-03-14 09:56:26.749133 | controller |     - '    logger.go:42: 09:44:51 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.749138 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.749144 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.749148 | controller |     - '        '
2026-03-14 09:56:26.749153 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.749157 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.749161 | controller |     - '        '
2026-03-14 09:56:26.749166 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.749170 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.749174 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.749179 | controller |     - '          exit 1'
2026-03-14 09:56:26.749183 | controller |     - '        fi'
2026-03-14 09:56:26.749187 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.749192 | controller |     - '        '
2026-03-14 09:56:26.749196 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.749200 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.749204 | controller |       | base64 -d)'
2026-03-14 09:56:26.749209 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.749213 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.749217 | controller |     - '          exit 1'
2026-03-14 09:56:26.749222 | controller |     - '        fi'
2026-03-14 09:56:26.749226 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.749230 | controller |     - '        '
2026-03-14 09:56:26.749234 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.749239 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.749243 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.749247 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.749251 | controller |       --timeout=300s'
2026-03-14 09:56:26.749255 | controller |     - '        '
2026-03-14 09:56:26.749260 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.749265 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.749270 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.749274 | controller |     - '        '
2026-03-14 09:56:26.749284 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.749289 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.749293 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.749297 | controller |     - '        '
2026-03-14 09:56:26.749302 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.749306 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.749310 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.749315 | controller |       available"'
2026-03-14 09:56:26.749321 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.749327 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.749337 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.749353 | controller |     - '          exit 1'
2026-03-14 09:56:26.749358 | controller |     - '        fi'
2026-03-14 09:56:26.749362 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.749366 | controller |     - '        '
2026-03-14 09:56:26.749371 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.749375 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.749379 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.749383 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.749388 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.749392 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.749400 | controller |       -- \'
2026-03-14 09:56:26.749404 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.749408 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.749412 | controller |     - '    logger.go:42: 09:44:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.749417 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.749421 | controller |     - '    logger.go:42: 09:44:51 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.749425 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.749429 | controller |       found'
2026-03-14 09:56:26.749434 | controller |     - '    logger.go:42: 09:44:52 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.749438 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.749442 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.749446 | controller |     - '        '
2026-03-14 09:56:26.749451 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.749455 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.749459 | controller |     - '        '
2026-03-14 09:56:26.749463 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.749468 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.749472 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.749476 | controller |     - '          exit 1'
2026-03-14 09:56:26.749481 | controller |     - '        fi'
2026-03-14 09:56:26.749485 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.749489 | controller |     - '        '
2026-03-14 09:56:26.749493 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.749498 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.749502 | controller |       | base64 -d)'
2026-03-14 09:56:26.749506 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.749510 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.749515 | controller |     - '          exit 1'
2026-03-14 09:56:26.749519 | controller |     - '        fi'
2026-03-14 09:56:26.749523 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.749527 | controller |     - '        '
2026-03-14 09:56:26.749532 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.749536 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.749540 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.749544 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.749552 | controller |       --timeout=300s'
2026-03-14 09:56:26.749557 | controller |     - '        '
2026-03-14 09:56:26.749561 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.749565 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.749569 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.749574 | controller |     - '        '
2026-03-14 09:56:26.749578 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.749589 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.749606 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.749612 | controller |     - '        '
2026-03-14 09:56:26.749617 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.749621 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.749626 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.749632 | controller |       available"'
2026-03-14 09:56:26.749638 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.749645 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.749650 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.749656 | controller |     - '          exit 1'
2026-03-14 09:56:26.749662 | controller |     - '        fi'
2026-03-14 09:56:26.749668 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.749674 | controller |     - '        '
2026-03-14 09:56:26.749679 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.749685 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.749691 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.749697 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.749703 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.749709 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.749715 | controller |       -- \'
2026-03-14 09:56:26.749721 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.749727 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.749733 | controller |     - '    logger.go:42: 09:44:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.749739 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.749745 | controller |     - '    logger.go:42: 09:44:52 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.749751 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.749757 | controller |       found'
2026-03-14 09:56:26.749763 | controller |     - '    logger.go:42: 09:44:53 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.749767 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.749771 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.749776 | controller |     - '        '
2026-03-14 09:56:26.749780 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.749784 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.749792 | controller |     - '        '
2026-03-14 09:56:26.749796 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.749800 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.749805 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.749809 | controller |     - '          exit 1'
2026-03-14 09:56:26.749813 | controller |     - '        fi'
2026-03-14 09:56:26.749820 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.749825 | controller |     - '        '
2026-03-14 09:56:26.749829 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.749833 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.749837 | controller |       | base64 -d)'
2026-03-14 09:56:26.749842 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.749846 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.749850 | controller |     - '          exit 1'
2026-03-14 09:56:26.749854 | controller |     - '        fi'
2026-03-14 09:56:26.749859 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.749863 | controller |     - '        '
2026-03-14 09:56:26.749867 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.749871 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.749875 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.749880 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.749884 | controller |       --timeout=300s'
2026-03-14 09:56:26.749888 | controller |     - '        '
2026-03-14 09:56:26.749892 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.749897 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.749901 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.749905 | controller |     - '        '
2026-03-14 09:56:26.749909 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.749924 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.749928 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.749935 | controller |     - '        '
2026-03-14 09:56:26.749939 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.749943 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.749948 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.749952 | controller |       available"'
2026-03-14 09:56:26.749956 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.749960 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.749965 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.749969 | controller |     - '          exit 1'
2026-03-14 09:56:26.749973 | controller |     - '        fi'
2026-03-14 09:56:26.749977 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.749982 | controller |     - '        '
2026-03-14 09:56:26.749986 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.749990 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.749995 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.750001 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.750005 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.750009 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.750013 | controller |       -- \'
2026-03-14 09:56:26.750018 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.750022 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.750026 | controller |     - '    logger.go:42: 09:44:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.750030 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.750035 | controller |     - '    logger.go:42: 09:44:54 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.750039 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.750043 | controller |       found'
2026-03-14 09:56:26.750048 | controller |     - '    logger.go:42: 09:44:55 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.750052 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.750056 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.750060 | controller |     - '        '
2026-03-14 09:56:26.750065 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.750069 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.750073 | controller |     - '        '
2026-03-14 09:56:26.750077 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.750082 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.750086 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.750090 | controller |     - '          exit 1'
2026-03-14 09:56:26.750094 | controller |     - '        fi'
2026-03-14 09:56:26.750099 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.750103 | controller |     - '        '
2026-03-14 09:56:26.750107 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.750111 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.750116 | controller |       | base64 -d)'
2026-03-14 09:56:26.750120 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.750124 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.750128 | controller |     - '          exit 1'
2026-03-14 09:56:26.750133 | controller |     - '        fi'
2026-03-14 09:56:26.750137 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.750141 | controller |     - '        '
2026-03-14 09:56:26.750145 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.750150 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.750154 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.750158 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.750162 | controller |       --timeout=300s'
2026-03-14 09:56:26.750166 | controller |     - '        '
2026-03-14 09:56:26.750171 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.750175 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.750179 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.750183 | controller |     - '        '
2026-03-14 09:56:26.750189 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.750193 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.750206 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.750210 | controller |     - '        '
2026-03-14 09:56:26.750215 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.750219 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.750223 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.750228 | controller |       available"'
2026-03-14 09:56:26.750232 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.750236 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.750240 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.750245 | controller |     - '          exit 1'
2026-03-14 09:56:26.750249 | controller |     - '        fi'
2026-03-14 09:56:26.750253 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.750257 | controller |     - '        '
2026-03-14 09:56:26.750262 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.750266 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.750270 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.750274 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.750279 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.750283 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.750287 | controller |       -- \'
2026-03-14 09:56:26.750291 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.750296 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.750300 | controller |     - '    logger.go:42: 09:44:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.750304 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.750309 | controller |     - '    logger.go:42: 09:44:55 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.750313 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.750317 | controller |       found'
2026-03-14 09:56:26.750321 | controller |     - '    logger.go:42: 09:44:56 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.750326 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.750330 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.750335 | controller |     - '        '
2026-03-14 09:56:26.750341 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.750348 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.750354 | controller |     - '        '
2026-03-14 09:56:26.750360 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.750366 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.750370 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.750374 | controller |     - '          exit 1'
2026-03-14 09:56:26.750378 | controller |     - '        fi'
2026-03-14 09:56:26.750383 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.750387 | controller |     - '        '
2026-03-14 09:56:26.750391 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.750397 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.750402 | controller |       | base64 -d)'
2026-03-14 09:56:26.750406 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.750410 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.750415 | controller |     - '          exit 1'
2026-03-14 09:56:26.750419 | controller |     - '        fi'
2026-03-14 09:56:26.750423 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.750427 | controller |     - '        '
2026-03-14 09:56:26.750432 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.750436 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.750440 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.750444 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.750449 | controller |       --timeout=300s'
2026-03-14 09:56:26.750453 | controller |     - '        '
2026-03-14 09:56:26.750457 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.750461 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.750466 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.750470 | controller |     - '        '
2026-03-14 09:56:26.750474 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.750478 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.750490 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.750494 | controller |     - '        '
2026-03-14 09:56:26.750499 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.750503 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.750507 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.750512 | controller |       available"'
2026-03-14 09:56:26.750516 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.750520 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.750524 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.750529 | controller |     - '          exit 1'
2026-03-14 09:56:26.750533 | controller |     - '        fi'
2026-03-14 09:56:26.750537 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.750541 | controller |     - '        '
2026-03-14 09:56:26.750546 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.750550 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.750554 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.750558 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.750563 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.750567 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.750571 | controller |       -- \'
2026-03-14 09:56:26.750575 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.750583 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.750587 | controller |     - '    logger.go:42: 09:44:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.750591 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.750614 | controller |     - '    logger.go:42: 09:44:56 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.750618 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.750622 | controller |       found'
2026-03-14 09:56:26.750627 | controller |     - '    logger.go:42: 09:44:57 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.750631 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.750635 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.750640 | controller |     - '        '
2026-03-14 09:56:26.750644 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.750648 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.750652 | controller |     - '        '
2026-03-14 09:56:26.750656 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.750661 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.750665 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.750669 | controller |     - '          exit 1'
2026-03-14 09:56:26.750673 | controller |     - '        fi'
2026-03-14 09:56:26.750678 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.750682 | controller |     - '        '
2026-03-14 09:56:26.750686 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.750693 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.750698 | controller |       | base64 -d)'
2026-03-14 09:56:26.750702 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.750706 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.750710 | controller |     - '          exit 1'
2026-03-14 09:56:26.750715 | controller |     - '        fi'
2026-03-14 09:56:26.750719 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.750723 | controller |     - '        '
2026-03-14 09:56:26.750727 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.750732 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.750736 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.750740 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.750744 | controller |       --timeout=300s'
2026-03-14 09:56:26.750748 | controller |     - '        '
2026-03-14 09:56:26.750753 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.750759 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.750763 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.750767 | controller |     - '        '
2026-03-14 09:56:26.750771 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.750776 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.750780 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.750791 | controller |     - '        '
2026-03-14 09:56:26.750796 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.750800 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.750806 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.750812 | controller |       available"'
2026-03-14 09:56:26.750816 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.750821 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.750825 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.750829 | controller |     - '          exit 1'
2026-03-14 09:56:26.750833 | controller |     - '        fi'
2026-03-14 09:56:26.750841 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.750845 | controller |     - '        '
2026-03-14 09:56:26.750850 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.750854 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.750858 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.750862 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.750866 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.750871 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.750875 | controller |       -- \'
2026-03-14 09:56:26.750879 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.750884 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.750888 | controller |     - '    logger.go:42: 09:44:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.750892 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.750896 | controller |     - '    logger.go:42: 09:44:57 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.750901 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.750905 | controller |       found'
2026-03-14 09:56:26.750909 | controller |     - '    logger.go:42: 09:44:58 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.750914 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.750918 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.750922 | controller |     - '        '
2026-03-14 09:56:26.750927 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.750931 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.750935 | controller |     - '        '
2026-03-14 09:56:26.750939 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.750944 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.750948 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.750952 | controller |     - '          exit 1'
2026-03-14 09:56:26.750956 | controller |     - '        fi'
2026-03-14 09:56:26.750961 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.750965 | controller |     - '        '
2026-03-14 09:56:26.750969 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.750973 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.750977 | controller |       | base64 -d)'
2026-03-14 09:56:26.750982 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.750986 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.750990 | controller |     - '          exit 1'
2026-03-14 09:56:26.750994 | controller |     - '        fi'
2026-03-14 09:56:26.750999 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.751005 | controller |     - '        '
2026-03-14 09:56:26.751026 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.751032 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.751037 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.751041 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.751045 | controller |       --timeout=300s'
2026-03-14 09:56:26.751050 | controller |     - '        '
2026-03-14 09:56:26.751054 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.751058 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.751065 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.751069 | controller |     - '        '
2026-03-14 09:56:26.751073 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.751078 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.751082 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.751086 | controller |     - '        '
2026-03-14 09:56:26.751097 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.751102 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.751106 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.751111 | controller |       available"'
2026-03-14 09:56:26.751115 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.751119 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.751123 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.751128 | controller |     - '          exit 1'
2026-03-14 09:56:26.751132 | controller |     - '        fi'
2026-03-14 09:56:26.751136 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.751140 | controller |     - '        '
2026-03-14 09:56:26.751144 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.751149 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.751153 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.751157 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.751161 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.751166 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.751170 | controller |       -- \'
2026-03-14 09:56:26.751174 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.751178 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.751183 | controller |     - '    logger.go:42: 09:44:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.751187 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.751191 | controller |     - '    logger.go:42: 09:44:58 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.751195 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.751199 | controller |       found'
2026-03-14 09:56:26.751206 | controller |     - '    logger.go:42: 09:44:59 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.751210 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.751214 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.751219 | controller |     - '        '
2026-03-14 09:56:26.751223 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.751227 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.751231 | controller |     - '        '
2026-03-14 09:56:26.751235 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.751240 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.751244 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.751248 | controller |     - '          exit 1'
2026-03-14 09:56:26.751252 | controller |     - '        fi'
2026-03-14 09:56:26.751257 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.751261 | controller |     - '        '
2026-03-14 09:56:26.751265 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.751269 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.751273 | controller |       | base64 -d)'
2026-03-14 09:56:26.751278 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.751282 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.751286 | controller |     - '          exit 1'
2026-03-14 09:56:26.751290 | controller |     - '        fi'
2026-03-14 09:56:26.751295 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.751299 | controller |     - '        '
2026-03-14 09:56:26.751303 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.751307 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.751312 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.751316 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.751320 | controller |       --timeout=300s'
2026-03-14 09:56:26.751324 | controller |     - '        '
2026-03-14 09:56:26.751328 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.751333 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.751337 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.751341 | controller |     - '        '
2026-03-14 09:56:26.751345 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.751350 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.751356 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.751363 | controller |     - '        '
2026-03-14 09:56:26.751371 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.751387 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.751392 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.751397 | controller |       available"'
2026-03-14 09:56:26.751401 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.751405 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.751410 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.751414 | controller |     - '          exit 1'
2026-03-14 09:56:26.751420 | controller |     - '        fi'
2026-03-14 09:56:26.751425 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.751431 | controller |     - '        '
2026-03-14 09:56:26.751435 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.751439 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.751444 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.751448 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.751452 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.751456 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.751461 | controller |       -- \'
2026-03-14 09:56:26.751465 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.751469 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.751473 | controller |     - '    logger.go:42: 09:44:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.751478 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.751482 | controller |     - '    logger.go:42: 09:44:59 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.751486 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.751490 | controller |       found'
2026-03-14 09:56:26.751495 | controller |     - '    logger.go:42: 09:45:00 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.751499 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.751503 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.751507 | controller |     - '        '
2026-03-14 09:56:26.751512 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.751518 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.751524 | controller |     - '        '
2026-03-14 09:56:26.751530 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.751536 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.751542 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.751548 | controller |     - '          exit 1'
2026-03-14 09:56:26.751554 | controller |     - '        fi'
2026-03-14 09:56:26.751558 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.751563 | controller |     - '        '
2026-03-14 09:56:26.751567 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.751571 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.751575 | controller |       | base64 -d)'
2026-03-14 09:56:26.751579 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.751584 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.751588 | controller |     - '          exit 1'
2026-03-14 09:56:26.751592 | controller |     - '        fi'
2026-03-14 09:56:26.751611 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.751616 | controller |     - '        '
2026-03-14 09:56:26.751620 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.751624 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.751629 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.751633 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.751637 | controller |       --timeout=300s'
2026-03-14 09:56:26.751644 | controller |     - '        '
2026-03-14 09:56:26.751648 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.751652 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.751657 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.751661 | controller |     - '        '
2026-03-14 09:56:26.751665 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.751669 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.751673 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.751678 | controller |     - '        '
2026-03-14 09:56:26.751682 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.751695 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.751699 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.751704 | controller |       available"'
2026-03-14 09:56:26.751708 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.751712 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.751716 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.751721 | controller |     - '          exit 1'
2026-03-14 09:56:26.751725 | controller |     - '        fi'
2026-03-14 09:56:26.751731 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.751736 | controller |     - '        '
2026-03-14 09:56:26.751740 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.751744 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.751748 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.751753 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.751757 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.751761 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.751765 | controller |       -- \'
2026-03-14 09:56:26.751770 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.751774 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.751778 | controller |     - '    logger.go:42: 09:45:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.751783 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.751788 | controller |     - '    logger.go:42: 09:45:01 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.751794 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.751799 | controller |       found'
2026-03-14 09:56:26.751806 | controller |     - '    logger.go:42: 09:45:02 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.751812 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.751818 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.751824 | controller |     - '        '
2026-03-14 09:56:26.751830 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.751836 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.751842 | controller |     - '        '
2026-03-14 09:56:26.751849 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.751853 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.751857 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.751862 | controller |     - '          exit 1'
2026-03-14 09:56:26.751866 | controller |     - '        fi'
2026-03-14 09:56:26.751870 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.751874 | controller |     - '        '
2026-03-14 09:56:26.751879 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.751883 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.751887 | controller |       | base64 -d)'
2026-03-14 09:56:26.751892 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.751896 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.751900 | controller |     - '          exit 1'
2026-03-14 09:56:26.751904 | controller |     - '        fi'
2026-03-14 09:56:26.751909 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.751913 | controller |     - '        '
2026-03-14 09:56:26.751917 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.751921 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.751926 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.751930 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.751934 | controller |       --timeout=300s'
2026-03-14 09:56:26.751941 | controller |     - '        '
2026-03-14 09:56:26.751945 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.751950 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.751954 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.751958 | controller |     - '        '
2026-03-14 09:56:26.751962 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.751967 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.751971 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.751975 | controller |     - '        '
2026-03-14 09:56:26.751979 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.751984 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.751996 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.752001 | controller |       available"'
2026-03-14 09:56:26.752005 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.752010 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.752016 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.752020 | controller |     - '          exit 1'
2026-03-14 09:56:26.752024 | controller |     - '        fi'
2026-03-14 09:56:26.752029 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.752033 | controller |     - '        '
2026-03-14 09:56:26.752037 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.752043 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.752048 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.752054 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.752060 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.752064 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.752069 | controller |       -- \'
2026-03-14 09:56:26.752073 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.752077 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.752081 | controller |     - '    logger.go:42: 09:45:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.752086 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.752090 | controller |     - '    logger.go:42: 09:45:02 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.752094 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.752098 | controller |       found'
2026-03-14 09:56:26.752103 | controller |     - '    logger.go:42: 09:45:03 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.752107 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.752111 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.752116 | controller |     - '        '
2026-03-14 09:56:26.752120 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.752124 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.752128 | controller |     - '        '
2026-03-14 09:56:26.752133 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.752137 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.752141 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.752145 | controller |     - '          exit 1'
2026-03-14 09:56:26.752150 | controller |     - '        fi'
2026-03-14 09:56:26.752154 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.752261 | controller |     - '        '
2026-03-14 09:56:26.752265 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.752270 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.752277 | controller |       | base64 -d)'
2026-03-14 09:56:26.752281 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.752285 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.752290 | controller |     - '          exit 1'
2026-03-14 09:56:26.752294 | controller |     - '        fi'
2026-03-14 09:56:26.752298 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.752302 | controller |     - '        '
2026-03-14 09:56:26.752307 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.752311 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.752315 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.752319 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.752324 | controller |       --timeout=300s'
2026-03-14 09:56:26.752328 | controller |     - '        '
2026-03-14 09:56:26.752333 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.752339 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.752344 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.752348 | controller |     - '        '
2026-03-14 09:56:26.752353 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.752359 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.752363 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.752367 | controller |     - '        '
2026-03-14 09:56:26.752372 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.752378 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.752385 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.752400 | controller |       available"'
2026-03-14 09:56:26.752405 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.752410 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.752414 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.752418 | controller |     - '          exit 1'
2026-03-14 09:56:26.752423 | controller |     - '        fi'
2026-03-14 09:56:26.752427 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.752431 | controller |     - '        '
2026-03-14 09:56:26.752435 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.752440 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.752444 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.752448 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.752452 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.752457 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.752461 | controller |       -- \'
2026-03-14 09:56:26.752465 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.752470 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.752474 | controller |     - '    logger.go:42: 09:45:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.752478 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.752482 | controller |     - '    logger.go:42: 09:45:03 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.752487 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.752491 | controller |       found'
2026-03-14 09:56:26.752495 | controller |     - '    logger.go:42: 09:45:04 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.752499 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.752504 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.752508 | controller |     - '        '
2026-03-14 09:56:26.752512 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.752516 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.752521 | controller |     - '        '
2026-03-14 09:56:26.752525 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.752529 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.752533 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.752538 | controller |     - '          exit 1'
2026-03-14 09:56:26.752542 | controller |     - '        fi'
2026-03-14 09:56:26.752546 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.752551 | controller |     - '        '
2026-03-14 09:56:26.752555 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.752561 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.752566 | controller |       | base64 -d)'
2026-03-14 09:56:26.752570 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.752574 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.752578 | controller |     - '          exit 1'
2026-03-14 09:56:26.752583 | controller |     - '        fi'
2026-03-14 09:56:26.752587 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.752591 | controller |     - '        '
2026-03-14 09:56:26.752610 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.752615 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.752619 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.752623 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.752628 | controller |       --timeout=300s'
2026-03-14 09:56:26.752632 | controller |     - '        '
2026-03-14 09:56:26.752636 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.752640 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.752645 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.752649 | controller |     - '        '
2026-03-14 09:56:26.752653 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.752658 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.752662 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.752666 | controller |     - '        '
2026-03-14 09:56:26.752670 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.752675 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.752679 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.752683 | controller |       available"'
2026-03-14 09:56:26.752694 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.752699 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.752703 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.752708 | controller |     - '          exit 1'
2026-03-14 09:56:26.752712 | controller |     - '        fi'
2026-03-14 09:56:26.752716 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.752720 | controller |     - '        '
2026-03-14 09:56:26.752725 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.752729 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.752733 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.752738 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.752744 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.752749 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.752753 | controller |       -- \'
2026-03-14 09:56:26.752757 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.752761 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.752769 | controller |     - '    logger.go:42: 09:45:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.752773 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.752777 | controller |     - '    logger.go:42: 09:45:04 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.752781 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.752786 | controller |       found'
2026-03-14 09:56:26.752790 | controller |     - '    logger.go:42: 09:45:05 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.752794 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.752798 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.752803 | controller |     - '        '
2026-03-14 09:56:26.752807 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.752811 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.752815 | controller |     - '        '
2026-03-14 09:56:26.752819 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.752824 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.752830 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.752834 | controller |     - '          exit 1'
2026-03-14 09:56:26.752838 | controller |     - '        fi'
2026-03-14 09:56:26.752843 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.752847 | controller |     - '        '
2026-03-14 09:56:26.752851 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.752856 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.752860 | controller |       | base64 -d)'
2026-03-14 09:56:26.752864 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.752868 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.752873 | controller |     - '          exit 1'
2026-03-14 09:56:26.752879 | controller |     - '        fi'
2026-03-14 09:56:26.752885 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.752891 | controller |     - '        '
2026-03-14 09:56:26.752897 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.752904 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.752910 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.752916 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.752921 | controller |       --timeout=300s'
2026-03-14 09:56:26.752927 | controller |     - '        '
2026-03-14 09:56:26.752933 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.752938 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.752944 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.752950 | controller |     - '        '
2026-03-14 09:56:26.752955 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.752961 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.752967 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.752973 | controller |     - '        '
2026-03-14 09:56:26.752979 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.752985 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.752994 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.753000 | controller |       available"'
2026-03-14 09:56:26.753006 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.753023 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.753030 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.753036 | controller |     - '          exit 1'
2026-03-14 09:56:26.753042 | controller |     - '        fi'
2026-03-14 09:56:26.753047 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.753053 | controller |     - '        '
2026-03-14 09:56:26.753059 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.753065 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.753070 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.753077 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.753083 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.753089 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.753095 | controller |       -- \'
2026-03-14 09:56:26.753101 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.753107 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.753113 | controller |     - '    logger.go:42: 09:45:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.753119 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.753126 | controller |     - '    logger.go:42: 09:45:05 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.753130 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.753135 | controller |       found'
2026-03-14 09:56:26.753139 | controller |     - '    logger.go:42: 09:45:06 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.753144 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.753148 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.753153 | controller |     - '        '
2026-03-14 09:56:26.753159 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.753164 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.753168 | controller |     - '        '
2026-03-14 09:56:26.753174 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.753180 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.753186 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.753192 | controller |     - '          exit 1'
2026-03-14 09:56:26.753197 | controller |     - '        fi'
2026-03-14 09:56:26.753203 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.753207 | controller |     - '        '
2026-03-14 09:56:26.753212 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.753216 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.753222 | controller |       | base64 -d)'
2026-03-14 09:56:26.753232 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.753237 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.753241 | controller |     - '          exit 1'
2026-03-14 09:56:26.753245 | controller |     - '        fi'
2026-03-14 09:56:26.753249 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.753256 | controller |     - '        '
2026-03-14 09:56:26.753262 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.753268 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.753274 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.753278 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.753283 | controller |       --timeout=300s'
2026-03-14 09:56:26.753287 | controller |     - '        '
2026-03-14 09:56:26.753291 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.753295 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.753300 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.753306 | controller |     - '        '
2026-03-14 09:56:26.753310 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.753315 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.753319 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.753323 | controller |     - '        '
2026-03-14 09:56:26.753329 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.753335 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.753341 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.753347 | controller |       available"'
2026-03-14 09:56:26.753353 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.753358 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.753375 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.753381 | controller |     - '          exit 1'
2026-03-14 09:56:26.753385 | controller |     - '        fi'
2026-03-14 09:56:26.753390 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.753396 | controller |     - '        '
2026-03-14 09:56:26.753402 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.753409 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.753415 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.753420 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.753424 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.753429 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.753433 | controller |       -- \'
2026-03-14 09:56:26.753437 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.753441 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.753449 | controller |     - '    logger.go:42: 09:45:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.753453 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.753457 | controller |     - '    logger.go:42: 09:45:06 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.753462 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.753468 | controller |       found'
2026-03-14 09:56:26.753474 | controller |     - '    logger.go:42: 09:45:07 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.753483 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.753487 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.753493 | controller |     - '        '
2026-03-14 09:56:26.753498 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.753504 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.753508 | controller |     - '        '
2026-03-14 09:56:26.753513 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.753517 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.753523 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.753529 | controller |     - '          exit 1'
2026-03-14 09:56:26.753534 | controller |     - '        fi'
2026-03-14 09:56:26.753538 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.753543 | controller |     - '        '
2026-03-14 09:56:26.753547 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.753555 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.753561 | controller |       | base64 -d)'
2026-03-14 09:56:26.753566 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.753570 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.753575 | controller |     - '          exit 1'
2026-03-14 09:56:26.753579 | controller |     - '        fi'
2026-03-14 09:56:26.753583 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.753588 | controller |     - '        '
2026-03-14 09:56:26.753592 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.753612 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.753617 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.753621 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.753626 | controller |       --timeout=300s'
2026-03-14 09:56:26.753630 | controller |     - '        '
2026-03-14 09:56:26.753634 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.753638 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.753642 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.753647 | controller |     - '        '
2026-03-14 09:56:26.753651 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.753655 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.753659 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.753664 | controller |     - '        '
2026-03-14 09:56:26.753668 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.753672 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.753676 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.753681 | controller |       available"'
2026-03-14 09:56:26.753685 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.753689 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.753701 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.753706 | controller |     - '          exit 1'
2026-03-14 09:56:26.753710 | controller |     - '        fi'
2026-03-14 09:56:26.753715 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.753722 | controller |     - '        '
2026-03-14 09:56:26.753727 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.753731 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.753735 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.753739 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.753744 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.753748 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.753755 | controller |       -- \'
2026-03-14 09:56:26.753760 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.753766 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.753772 | controller |     - '    logger.go:42: 09:45:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.753777 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.753781 | controller |     - '    logger.go:42: 09:45:08 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.753786 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.753792 | controller |       found'
2026-03-14 09:56:26.753798 | controller |     - '    logger.go:42: 09:45:09 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.753803 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.753807 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.753811 | controller |     - '        '
2026-03-14 09:56:26.753817 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.753823 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.753828 | controller |     - '        '
2026-03-14 09:56:26.753833 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.753837 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.753841 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.753845 | controller |     - '          exit 1'
2026-03-14 09:56:26.753851 | controller |     - '        fi'
2026-03-14 09:56:26.753857 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.753863 | controller |     - '        '
2026-03-14 09:56:26.753867 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.753872 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.753876 | controller |       | base64 -d)'
2026-03-14 09:56:26.753880 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.753885 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.753889 | controller |     - '          exit 1'
2026-03-14 09:56:26.753893 | controller |     - '        fi'
2026-03-14 09:56:26.753897 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.753902 | controller |     - '        '
2026-03-14 09:56:26.753906 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.753910 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.753914 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.753919 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.753923 | controller |       --timeout=300s'
2026-03-14 09:56:26.753927 | controller |     - '        '
2026-03-14 09:56:26.753931 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.753939 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.753944 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.753948 | controller |     - '        '
2026-03-14 09:56:26.753952 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.753956 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.753961 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.753965 | controller |     - '        '
2026-03-14 09:56:26.753969 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.753973 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.753978 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.753982 | controller |       available"'
2026-03-14 09:56:26.753986 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.753990 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.753995 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.753999 | controller |     - '          exit 1'
2026-03-14 09:56:26.754010 | controller |     - '        fi'
2026-03-14 09:56:26.754015 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.754019 | controller |     - '        '
2026-03-14 09:56:26.754024 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.754028 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.754032 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.754036 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.754040 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.754045 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.754049 | controller |       -- \'
2026-03-14 09:56:26.754053 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.754057 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.754063 | controller |     - '    logger.go:42: 09:45:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.754069 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.754075 | controller |     - '    logger.go:42: 09:45:09 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.754083 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.754087 | controller |       found'
2026-03-14 09:56:26.754093 | controller |     - '    logger.go:42: 09:45:10 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.754099 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.754104 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.754108 | controller |     - '        '
2026-03-14 09:56:26.754113 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.754119 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.754125 | controller |     - '        '
2026-03-14 09:56:26.754130 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.754137 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.754142 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.754146 | controller |     - '          exit 1'
2026-03-14 09:56:26.754151 | controller |     - '        fi'
2026-03-14 09:56:26.754157 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.754163 | controller |     - '        '
2026-03-14 09:56:26.754167 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.754171 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.754176 | controller |       | base64 -d)'
2026-03-14 09:56:26.754180 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.754184 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.754188 | controller |     - '          exit 1'
2026-03-14 09:56:26.754193 | controller |     - '        fi'
2026-03-14 09:56:26.754197 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.754201 | controller |     - '        '
2026-03-14 09:56:26.754206 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.754210 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.754214 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.754218 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.754222 | controller |       --timeout=300s'
2026-03-14 09:56:26.754227 | controller |     - '        '
2026-03-14 09:56:26.754231 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.754235 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.754240 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.754244 | controller |     - '        '
2026-03-14 09:56:26.754248 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.754252 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.754257 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.754261 | controller |     - '        '
2026-03-14 09:56:26.754265 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.754269 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.754274 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.754278 | controller |       available"'
2026-03-14 09:56:26.754282 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.754286 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.754291 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.754295 | controller |     - '          exit 1'
2026-03-14 09:56:26.754299 | controller |     - '        fi'
2026-03-14 09:56:26.754311 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.754315 | controller |     - '        '
2026-03-14 09:56:26.754320 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.754324 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.754329 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.754335 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.754342 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.754350 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.754356 | controller |       -- \'
2026-03-14 09:56:26.754363 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.754370 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.754375 | controller |     - '    logger.go:42: 09:45:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.754379 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.754385 | controller |     - '    logger.go:42: 09:45:10 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.754391 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.754396 | controller |       found'
2026-03-14 09:56:26.754402 | controller |     - '    logger.go:42: 09:45:11 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.754408 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.754414 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.754420 | controller |     - '        '
2026-03-14 09:56:26.754426 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.754432 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.754439 | controller |     - '        '
2026-03-14 09:56:26.754445 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.754451 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.754456 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.754461 | controller |     - '          exit 1'
2026-03-14 09:56:26.754465 | controller |     - '        fi'
2026-03-14 09:56:26.754469 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.754473 | controller |     - '        '
2026-03-14 09:56:26.754478 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.754482 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.754486 | controller |       | base64 -d)'
2026-03-14 09:56:26.754490 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.754495 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.754499 | controller |     - '          exit 1'
2026-03-14 09:56:26.754503 | controller |     - '        fi'
2026-03-14 09:56:26.754507 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.754512 | controller |     - '        '
2026-03-14 09:56:26.754516 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.754520 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.754524 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.754529 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.754533 | controller |       --timeout=300s'
2026-03-14 09:56:26.754537 | controller |     - '        '
2026-03-14 09:56:26.754541 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.754546 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.754550 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.754554 | controller |     - '        '
2026-03-14 09:56:26.754558 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.754566 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.754570 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.754575 | controller |     - '        '
2026-03-14 09:56:26.754579 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.754583 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.754587 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.754591 | controller |       available"'
2026-03-14 09:56:26.754610 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.754615 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.754619 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.754623 | controller |     - '          exit 1'
2026-03-14 09:56:26.754627 | controller |     - '        fi'
2026-03-14 09:56:26.754632 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.754643 | controller |     - '        '
2026-03-14 09:56:26.754648 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.754652 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.754657 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.754661 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.754665 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.754670 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.754674 | controller |       -- \'
2026-03-14 09:56:26.754678 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.754682 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.754687 | controller |     - '    logger.go:42: 09:45:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.754691 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.754695 | controller |     - '    logger.go:42: 09:45:11 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.754700 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.754707 | controller |       found'
2026-03-14 09:56:26.754712 | controller |     - '    logger.go:42: 09:45:12 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.754716 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.754720 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.754724 | controller |     - '        '
2026-03-14 09:56:26.754729 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.754733 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.754737 | controller |     - '        '
2026-03-14 09:56:26.754741 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.754746 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.754750 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.754754 | controller |     - '          exit 1'
2026-03-14 09:56:26.754758 | controller |     - '        fi'
2026-03-14 09:56:26.754763 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.754767 | controller |     - '        '
2026-03-14 09:56:26.754771 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.754775 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.754782 | controller |       | base64 -d)'
2026-03-14 09:56:26.754786 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.754790 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.754794 | controller |     - '          exit 1'
2026-03-14 09:56:26.754799 | controller |     - '        fi'
2026-03-14 09:56:26.754803 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.754807 | controller |     - '        '
2026-03-14 09:56:26.754811 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.754816 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.754820 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.754824 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.754828 | controller |       --timeout=300s'
2026-03-14 09:56:26.754833 | controller |     - '        '
2026-03-14 09:56:26.754837 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.754841 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.754845 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.754850 | controller |     - '        '
2026-03-14 09:56:26.754854 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.754858 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.754862 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.754867 | controller |     - '        '
2026-03-14 09:56:26.754871 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.754875 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.754879 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.754884 | controller |       available"'
2026-03-14 09:56:26.754888 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.754892 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.754898 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.754902 | controller |     - '          exit 1'
2026-03-14 09:56:26.754907 | controller |     - '        fi'
2026-03-14 09:56:26.754911 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.754915 | controller |     - '        '
2026-03-14 09:56:26.754926 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.754930 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.754934 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.754939 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.754943 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.754947 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.754951 | controller |       -- \'
2026-03-14 09:56:26.754956 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.754960 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.754966 | controller |     - '    logger.go:42: 09:45:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.754970 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.754974 | controller |     - '    logger.go:42: 09:45:12 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.754979 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.754983 | controller |       found'
2026-03-14 09:56:26.754987 | controller |     - '    logger.go:42: 09:45:13 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.754991 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.754996 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.755000 | controller |     - '        '
2026-03-14 09:56:26.755004 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.755008 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.755031 | controller |     - '        '
2026-03-14 09:56:26.755036 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.755040 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.755044 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.755048 | controller |     - '          exit 1'
2026-03-14 09:56:26.755053 | controller |     - '        fi'
2026-03-14 09:56:26.755057 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.755061 | controller |     - '        '
2026-03-14 09:56:26.755065 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.755070 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.755074 | controller |       | base64 -d)'
2026-03-14 09:56:26.755078 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.755082 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.755087 | controller |     - '          exit 1'
2026-03-14 09:56:26.755091 | controller |     - '        fi'
2026-03-14 09:56:26.755095 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.755100 | controller |     - '        '
2026-03-14 09:56:26.755104 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.755108 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.755112 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.755117 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.755121 | controller |       --timeout=300s'
2026-03-14 09:56:26.755125 | controller |     - '        '
2026-03-14 09:56:26.755129 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.755136 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.755140 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.755144 | controller |     - '        '
2026-03-14 09:56:26.755149 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.755153 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.755157 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.755162 | controller |     - '        '
2026-03-14 09:56:26.755166 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.755170 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.755174 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.755180 | controller |       available"'
2026-03-14 09:56:26.755185 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.755189 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.755193 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.755197 | controller |     - '          exit 1'
2026-03-14 09:56:26.755202 | controller |     - '        fi'
2026-03-14 09:56:26.755206 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.755210 | controller |     - '        '
2026-03-14 09:56:26.755214 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.755226 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.755230 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.755235 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.755239 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.755243 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.755247 | controller |       -- \'
2026-03-14 09:56:26.755252 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.755256 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.755260 | controller |     - '    logger.go:42: 09:45:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.755264 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.755269 | controller |     - '    logger.go:42: 09:45:13 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.755273 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.755277 | controller |       found'
2026-03-14 09:56:26.755281 | controller |     - '    logger.go:42: 09:45:14 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.755287 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.755292 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.755296 | controller |     - '        '
2026-03-14 09:56:26.755300 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.755304 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.755308 | controller |     - '        '
2026-03-14 09:56:26.755313 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.755317 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.755321 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.755325 | controller |     - '          exit 1'
2026-03-14 09:56:26.755330 | controller |     - '        fi'
2026-03-14 09:56:26.755334 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.755338 | controller |     - '        '
2026-03-14 09:56:26.755342 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.755346 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.755351 | controller |       | base64 -d)'
2026-03-14 09:56:26.755355 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.755359 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.755363 | controller |     - '          exit 1'
2026-03-14 09:56:26.755367 | controller |     - '        fi'
2026-03-14 09:56:26.755372 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.755376 | controller |     - '        '
2026-03-14 09:56:26.755380 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.755388 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.755392 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.755396 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.755400 | controller |       --timeout=300s'
2026-03-14 09:56:26.755405 | controller |     - '        '
2026-03-14 09:56:26.755409 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.755413 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.755417 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.755422 | controller |     - '        '
2026-03-14 09:56:26.755426 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.755430 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.755438 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.755454 | controller |     - '        '
2026-03-14 09:56:26.755461 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.755466 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.755470 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.755475 | controller |       available"'
2026-03-14 09:56:26.755479 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.755483 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.755487 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.755492 | controller |     - '          exit 1'
2026-03-14 09:56:26.755496 | controller |     - '        fi'
2026-03-14 09:56:26.755500 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.755504 | controller |     - '        '
2026-03-14 09:56:26.755509 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.755520 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.755525 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.755529 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.755533 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.755537 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.755542 | controller |       -- \'
2026-03-14 09:56:26.755546 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.755550 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.755554 | controller |     - '    logger.go:42: 09:45:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.755559 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.755563 | controller |     - '    logger.go:42: 09:45:14 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.755567 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.755571 | controller |       found'
2026-03-14 09:56:26.755576 | controller |     - '    logger.go:42: 09:45:15 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.755582 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.755586 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.755590 | controller |     - '        '
2026-03-14 09:56:26.755606 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.755613 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.755617 | controller |     - '        '
2026-03-14 09:56:26.755621 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.755625 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.755630 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.755634 | controller |     - '          exit 1'
2026-03-14 09:56:26.755638 | controller |     - '        fi'
2026-03-14 09:56:26.755642 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.755647 | controller |     - '        '
2026-03-14 09:56:26.755651 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.755655 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.755659 | controller |       | base64 -d)'
2026-03-14 09:56:26.755664 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.755668 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.755672 | controller |     - '          exit 1'
2026-03-14 09:56:26.755676 | controller |     - '        fi'
2026-03-14 09:56:26.755681 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.755685 | controller |     - '        '
2026-03-14 09:56:26.755689 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.755693 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.755698 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.755702 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.755706 | controller |       --timeout=300s'
2026-03-14 09:56:26.755710 | controller |     - '        '
2026-03-14 09:56:26.755714 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.755719 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.755723 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.755727 | controller |     - '        '
2026-03-14 09:56:26.755731 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.755736 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.755740 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.755744 | controller |     - '        '
2026-03-14 09:56:26.755749 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.755753 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.755757 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.755761 | controller |       available"'
2026-03-14 09:56:26.755766 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.755770 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.755774 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.755778 | controller |     - '          exit 1'
2026-03-14 09:56:26.755782 | controller |     - '        fi'
2026-03-14 09:56:26.755787 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.755793 | controller |     - '        '
2026-03-14 09:56:26.755797 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.755801 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.755812 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.755817 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.755821 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.755826 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.755830 | controller |       -- \'
2026-03-14 09:56:26.755834 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.755838 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.755843 | controller |     - '    logger.go:42: 09:45:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.755847 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.755851 | controller |     - '    logger.go:42: 09:45:16 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.755855 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.755860 | controller |       found'
2026-03-14 09:56:26.755864 | controller |     - '    logger.go:42: 09:45:17 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.755868 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.755873 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.755877 | controller |     - '        '
2026-03-14 09:56:26.755881 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.755888 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.755892 | controller |     - '        '
2026-03-14 09:56:26.755896 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.755900 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.755905 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.755909 | controller |     - '          exit 1'
2026-03-14 09:56:26.755913 | controller |     - '        fi'
2026-03-14 09:56:26.755917 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.755922 | controller |     - '        '
2026-03-14 09:56:26.755926 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.755930 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.755934 | controller |       | base64 -d)'
2026-03-14 09:56:26.755939 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.755943 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.755947 | controller |     - '          exit 1'
2026-03-14 09:56:26.755951 | controller |     - '        fi'
2026-03-14 09:56:26.755955 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.755960 | controller |     - '        '
2026-03-14 09:56:26.755964 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.755968 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.755972 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.755977 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.755981 | controller |       --timeout=300s'
2026-03-14 09:56:26.755985 | controller |     - '        '
2026-03-14 09:56:26.755989 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.755995 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.756000 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.756004 | controller |     - '        '
2026-03-14 09:56:26.756008 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.756012 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.756017 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.756021 | controller |     - '        '
2026-03-14 09:56:26.756025 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.756029 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.756034 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.756038 | controller |       available"'
2026-03-14 09:56:26.756042 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.756046 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.756051 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.756055 | controller |     - '          exit 1'
2026-03-14 09:56:26.756059 | controller |     - '        fi'
2026-03-14 09:56:26.756063 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.756068 | controller |     - '        '
2026-03-14 09:56:26.756072 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.756076 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.756086 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.756091 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.756095 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.756099 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.756104 | controller |       -- \'
2026-03-14 09:56:26.756108 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.756112 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.756117 | controller |     - '    logger.go:42: 09:45:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.756121 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.756125 | controller |     - '    logger.go:42: 09:45:17 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.756129 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.756134 | controller |       found'
2026-03-14 09:56:26.756138 | controller |     - '    logger.go:42: 09:45:18 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.756142 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.756146 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.756151 | controller |     - '        '
2026-03-14 09:56:26.756155 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.756159 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.756163 | controller |     - '        '
2026-03-14 09:56:26.756168 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.756172 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.756178 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.756182 | controller |     - '          exit 1'
2026-03-14 09:56:26.756186 | controller |     - '        fi'
2026-03-14 09:56:26.756191 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.756195 | controller |     - '        '
2026-03-14 09:56:26.756199 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.756206 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.756210 | controller |       | base64 -d)'
2026-03-14 09:56:26.756214 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.756218 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.756223 | controller |     - '          exit 1'
2026-03-14 09:56:26.756227 | controller |     - '        fi'
2026-03-14 09:56:26.756231 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.756236 | controller |     - '        '
2026-03-14 09:56:26.756240 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.756244 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.756248 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.756252 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.756257 | controller |       --timeout=300s'
2026-03-14 09:56:26.756261 | controller |     - '        '
2026-03-14 09:56:26.756265 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.756270 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.756274 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.756278 | controller |     - '        '
2026-03-14 09:56:26.756282 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.756287 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.756291 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.756295 | controller |     - '        '
2026-03-14 09:56:26.756299 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.756304 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.756308 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.756312 | controller |       available"'
2026-03-14 09:56:26.756316 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.756321 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.756325 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.756329 | controller |     - '          exit 1'
2026-03-14 09:56:26.756334 | controller |     - '        fi'
2026-03-14 09:56:26.756338 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.756342 | controller |     - '        '
2026-03-14 09:56:26.756346 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.756351 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.756355 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.756365 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.756370 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.756376 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.756380 | controller |       -- \'
2026-03-14 09:56:26.756385 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.756391 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.756396 | controller |     - '    logger.go:42: 09:45:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.756400 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.756404 | controller |     - '    logger.go:42: 09:45:18 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.756408 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.756413 | controller |       found'
2026-03-14 09:56:26.756417 | controller |     - '    logger.go:42: 09:45:19 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.756421 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.756425 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.756430 | controller |     - '        '
2026-03-14 09:56:26.756434 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.756438 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.756450 | controller |     - '        '
2026-03-14 09:56:26.756456 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.756462 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.756469 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.756475 | controller |     - '          exit 1'
2026-03-14 09:56:26.756481 | controller |     - '        fi'
2026-03-14 09:56:26.756485 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.756492 | controller |     - '        '
2026-03-14 09:56:26.756496 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.756500 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.756506 | controller |       | base64 -d)'
2026-03-14 09:56:26.756510 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.756515 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.756519 | controller |     - '          exit 1'
2026-03-14 09:56:26.756523 | controller |     - '        fi'
2026-03-14 09:56:26.756529 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.756534 | controller |     - '        '
2026-03-14 09:56:26.756538 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.756542 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.756547 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.756551 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.756555 | controller |       --timeout=300s'
2026-03-14 09:56:26.756559 | controller |     - '        '
2026-03-14 09:56:26.756564 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.756568 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.756572 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.756577 | controller |     - '        '
2026-03-14 09:56:26.756581 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.756585 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.756604 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.756611 | controller |     - '        '
2026-03-14 09:56:26.756615 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.756619 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.756624 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.756628 | controller |       available"'
2026-03-14 09:56:26.756632 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.756636 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.756641 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.756645 | controller |     - '          exit 1'
2026-03-14 09:56:26.756649 | controller |     - '        fi'
2026-03-14 09:56:26.756653 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.756658 | controller |     - '        '
2026-03-14 09:56:26.756662 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.756666 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.756670 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.756682 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.756688 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.756694 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.756699 | controller |       -- \'
2026-03-14 09:56:26.756704 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.756708 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.756712 | controller |     - '    logger.go:42: 09:45:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.756717 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.756721 | controller |     - '    logger.go:42: 09:45:19 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.756725 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.756729 | controller |       found'
2026-03-14 09:56:26.756734 | controller |     - '    logger.go:42: 09:45:20 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.756738 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.756745 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.756749 | controller |     - '        '
2026-03-14 09:56:26.756753 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.756758 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.756762 | controller |     - '        '
2026-03-14 09:56:26.756766 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.756771 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.756775 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.756779 | controller |     - '          exit 1'
2026-03-14 09:56:26.756784 | controller |     - '        fi'
2026-03-14 09:56:26.756788 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.756792 | controller |     - '        '
2026-03-14 09:56:26.756796 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.756801 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.756808 | controller |       | base64 -d)'
2026-03-14 09:56:26.756813 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.756817 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.756821 | controller |     - '          exit 1'
2026-03-14 09:56:26.756826 | controller |     - '        fi'
2026-03-14 09:56:26.756830 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.756834 | controller |     - '        '
2026-03-14 09:56:26.756838 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.756843 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.756847 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.756851 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.756855 | controller |       --timeout=300s'
2026-03-14 09:56:26.756860 | controller |     - '        '
2026-03-14 09:56:26.756864 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.756868 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.756872 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.756877 | controller |     - '        '
2026-03-14 09:56:26.756881 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.756885 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.756889 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.756894 | controller |     - '        '
2026-03-14 09:56:26.756898 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.756902 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.756906 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.756911 | controller |       available"'
2026-03-14 09:56:26.756915 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.756919 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.756923 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.756928 | controller |     - '          exit 1'
2026-03-14 09:56:26.756932 | controller |     - '        fi'
2026-03-14 09:56:26.756936 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.756940 | controller |     - '        '
2026-03-14 09:56:26.756945 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.756949 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.756953 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.756957 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.756968 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.756973 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.756977 | controller |       -- \'
2026-03-14 09:56:26.756982 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.756986 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.756990 | controller |     - '    logger.go:42: 09:45:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.756996 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.757000 | controller |     - '    logger.go:42: 09:45:20 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.757005 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.757009 | controller |       found'
2026-03-14 09:56:26.757013 | controller |     - '    logger.go:42: 09:45:21 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.757017 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.757022 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.757026 | controller |     - '        '
2026-03-14 09:56:26.757030 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.757034 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.757039 | controller |     - '        '
2026-03-14 09:56:26.757043 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.757049 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.757053 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.757057 | controller |     - '          exit 1'
2026-03-14 09:56:26.757062 | controller |     - '        fi'
2026-03-14 09:56:26.757066 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.757070 | controller |     - '        '
2026-03-14 09:56:26.757074 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.757079 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.757083 | controller |       | base64 -d)'
2026-03-14 09:56:26.757087 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.757091 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.757096 | controller |     - '          exit 1'
2026-03-14 09:56:26.757100 | controller |     - '        fi'
2026-03-14 09:56:26.757104 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.757109 | controller |     - '        '
2026-03-14 09:56:26.757113 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.757117 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.757121 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.757126 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.757130 | controller |       --timeout=300s'
2026-03-14 09:56:26.757134 | controller |     - '        '
2026-03-14 09:56:26.757138 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.757143 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.757147 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.757151 | controller |     - '        '
2026-03-14 09:56:26.757156 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.757160 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.757164 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.757168 | controller |     - '        '
2026-03-14 09:56:26.757173 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.757177 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.757181 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.757187 | controller |       available"'
2026-03-14 09:56:26.757191 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.757196 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.757200 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.757204 | controller |     - '          exit 1'
2026-03-14 09:56:26.757208 | controller |     - '        fi'
2026-03-14 09:56:26.757213 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.757217 | controller |     - '        '
2026-03-14 09:56:26.757221 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.757225 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.757230 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.757234 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.757244 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.757248 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.757253 | controller |       -- \'
2026-03-14 09:56:26.757257 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.757261 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.757266 | controller |     - '    logger.go:42: 09:45:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.757270 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.757274 | controller |     - '    logger.go:42: 09:45:22 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.757279 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.757283 | controller |       found'
2026-03-14 09:56:26.757287 | controller |     - '    logger.go:42: 09:45:23 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.757291 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.757296 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.757300 | controller |     - '        '
2026-03-14 09:56:26.757304 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.757308 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.757312 | controller |     - '        '
2026-03-14 09:56:26.757317 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.757321 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.757325 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.757329 | controller |     - '          exit 1'
2026-03-14 09:56:26.757334 | controller |     - '        fi'
2026-03-14 09:56:26.757338 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.757342 | controller |     - '        '
2026-03-14 09:56:26.757347 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.757351 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.757355 | controller |       | base64 -d)'
2026-03-14 09:56:26.757359 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.757364 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.757368 | controller |     - '          exit 1'
2026-03-14 09:56:26.757372 | controller |     - '        fi'
2026-03-14 09:56:26.757376 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.757381 | controller |     - '        '
2026-03-14 09:56:26.757385 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.757389 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.757395 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.757399 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.757404 | controller |       --timeout=300s'
2026-03-14 09:56:26.757408 | controller |     - '        '
2026-03-14 09:56:26.757412 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.757417 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.757421 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.757425 | controller |     - '        '
2026-03-14 09:56:26.757429 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.757434 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.757438 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.757444 | controller |     - '        '
2026-03-14 09:56:26.757449 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.757455 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.757460 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.757464 | controller |       available"'
2026-03-14 09:56:26.757469 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.757475 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.757485 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.757491 | controller |     - '          exit 1'
2026-03-14 09:56:26.757497 | controller |     - '        fi'
2026-03-14 09:56:26.757503 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.757509 | controller |     - '        '
2026-03-14 09:56:26.757515 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.757521 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.757527 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.757533 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.757537 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.757550 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.757554 | controller |       -- \'
2026-03-14 09:56:26.757559 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.757563 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.757567 | controller |     - '    logger.go:42: 09:45:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.757572 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.757576 | controller |     - '    logger.go:42: 09:45:23 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.757580 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.757584 | controller |       found'
2026-03-14 09:56:26.757589 | controller |     - '    logger.go:42: 09:45:24 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.757605 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.757612 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.757620 | controller |     - '        '
2026-03-14 09:56:26.757624 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.757628 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.757633 | controller |     - '        '
2026-03-14 09:56:26.757637 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.757641 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.757646 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.757650 | controller |     - '          exit 1'
2026-03-14 09:56:26.757657 | controller |     - '        fi'
2026-03-14 09:56:26.757661 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.757666 | controller |     - '        '
2026-03-14 09:56:26.757670 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.757674 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.757678 | controller |       | base64 -d)'
2026-03-14 09:56:26.757683 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.757688 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.757694 | controller |     - '          exit 1'
2026-03-14 09:56:26.757700 | controller |     - '        fi'
2026-03-14 09:56:26.757706 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.757712 | controller |     - '        '
2026-03-14 09:56:26.757718 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.757724 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.757733 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.757739 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.757744 | controller |       --timeout=300s'
2026-03-14 09:56:26.757749 | controller |     - '        '
2026-03-14 09:56:26.757753 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.757757 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.757762 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.757766 | controller |     - '        '
2026-03-14 09:56:26.757770 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.757774 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.757779 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.757783 | controller |     - '        '
2026-03-14 09:56:26.757787 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.757792 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.757796 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.757800 | controller |       available"'
2026-03-14 09:56:26.757805 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.757809 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.757813 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.757818 | controller |     - '          exit 1'
2026-03-14 09:56:26.757822 | controller |     - '        fi'
2026-03-14 09:56:26.757826 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.757830 | controller |     - '        '
2026-03-14 09:56:26.757837 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.757842 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.757847 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.757853 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.757858 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.757871 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.757878 | controller |       -- \'
2026-03-14 09:56:26.757882 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.757887 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.757891 | controller |     - '    logger.go:42: 09:45:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.757895 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.757899 | controller |     - '    logger.go:42: 09:45:24 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.757904 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.757908 | controller |       found'
2026-03-14 09:56:26.757912 | controller |     - '    logger.go:42: 09:45:25 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.757916 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.757920 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.757925 | controller |     - '        '
2026-03-14 09:56:26.757929 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.757933 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.757937 | controller |     - '        '
2026-03-14 09:56:26.757942 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.757946 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.757950 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.757954 | controller |     - '          exit 1'
2026-03-14 09:56:26.757959 | controller |     - '        fi'
2026-03-14 09:56:26.757963 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.757967 | controller |     - '        '
2026-03-14 09:56:26.757971 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.757976 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.757980 | controller |       | base64 -d)'
2026-03-14 09:56:26.757984 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.757988 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.757992 | controller |     - '          exit 1'
2026-03-14 09:56:26.757997 | controller |     - '        fi'
2026-03-14 09:56:26.758001 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.758005 | controller |     - '        '
2026-03-14 09:56:26.758009 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.758014 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.758018 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.758022 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.758026 | controller |       --timeout=300s'
2026-03-14 09:56:26.758030 | controller |     - '        '
2026-03-14 09:56:26.758035 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.758039 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.758047 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.758051 | controller |     - '        '
2026-03-14 09:56:26.758055 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.758060 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.758064 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.758068 | controller |     - '        '
2026-03-14 09:56:26.758072 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.758077 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.758081 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.758085 | controller |       available"'
2026-03-14 09:56:26.758089 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.758094 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.758098 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.758102 | controller |     - '          exit 1'
2026-03-14 09:56:26.758106 | controller |     - '        fi'
2026-03-14 09:56:26.758111 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.758115 | controller |     - '        '
2026-03-14 09:56:26.758119 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.758123 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.758128 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.758132 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.758136 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.758140 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.758145 | controller |       -- \'
2026-03-14 09:56:26.758157 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.758163 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.758169 | controller |     - '    logger.go:42: 09:45:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.758175 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.758181 | controller |     - '    logger.go:42: 09:45:25 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.758187 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.758193 | controller |       found'
2026-03-14 09:56:26.758199 | controller |     - '    logger.go:42: 09:45:26 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.758205 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.758211 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.758217 | controller |     - '        '
2026-03-14 09:56:26.758222 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.758228 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.758233 | controller |     - '        '
2026-03-14 09:56:26.758239 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.758244 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.758250 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.758258 | controller |     - '          exit 1'
2026-03-14 09:56:26.758263 | controller |     - '        fi'
2026-03-14 09:56:26.758267 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.758272 | controller |     - '        '
2026-03-14 09:56:26.758276 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.758280 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.758284 | controller |       | base64 -d)'
2026-03-14 09:56:26.758289 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.758293 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.758297 | controller |     - '          exit 1'
2026-03-14 09:56:26.758301 | controller |     - '        fi'
2026-03-14 09:56:26.758306 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.758310 | controller |     - '        '
2026-03-14 09:56:26.758314 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.758318 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.758323 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.758329 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.758334 | controller |       --timeout=300s'
2026-03-14 09:56:26.758338 | controller |     - '        '
2026-03-14 09:56:26.758342 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.758346 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.758351 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.758355 | controller |     - '        '
2026-03-14 09:56:26.758359 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.758363 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.758368 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.758380 | controller |     - '        '
2026-03-14 09:56:26.758384 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.758393 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.758397 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.758401 | controller |       available"'
2026-03-14 09:56:26.758405 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.758410 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.758414 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.758418 | controller |     - '          exit 1'
2026-03-14 09:56:26.758422 | controller |     - '        fi'
2026-03-14 09:56:26.758427 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.758431 | controller |     - '        '
2026-03-14 09:56:26.758435 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.758439 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.758444 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.758448 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.758452 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.758458 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.758463 | controller |       -- \'
2026-03-14 09:56:26.758476 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.758481 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.758487 | controller |     - '    logger.go:42: 09:45:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.758492 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.758499 | controller |     - '    logger.go:42: 09:45:26 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.758505 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.758511 | controller |       found'
2026-03-14 09:56:26.758517 | controller |     - '    logger.go:42: 09:45:27 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.758523 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.758528 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.758534 | controller |     - '        '
2026-03-14 09:56:26.758540 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.758545 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.758550 | controller |     - '        '
2026-03-14 09:56:26.758556 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.758561 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.758567 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.758571 | controller |     - '          exit 1'
2026-03-14 09:56:26.758576 | controller |     - '        fi'
2026-03-14 09:56:26.758580 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.758584 | controller |     - '        '
2026-03-14 09:56:26.758589 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.758605 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.758612 | controller |       | base64 -d)'
2026-03-14 09:56:26.758617 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.758621 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.758625 | controller |     - '          exit 1'
2026-03-14 09:56:26.758630 | controller |     - '        fi'
2026-03-14 09:56:26.758634 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.758638 | controller |     - '        '
2026-03-14 09:56:26.758642 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.758648 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.758654 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.758659 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.758665 | controller |       --timeout=300s'
2026-03-14 09:56:26.758670 | controller |     - '        '
2026-03-14 09:56:26.758676 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.758682 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.758692 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.758697 | controller |     - '        '
2026-03-14 09:56:26.758703 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.758708 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.758717 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.758722 | controller |     - '        '
2026-03-14 09:56:26.758727 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.758733 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.758739 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.758744 | controller |       available"'
2026-03-14 09:56:26.758749 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.758754 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.758758 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.758762 | controller |     - '          exit 1'
2026-03-14 09:56:26.758769 | controller |     - '        fi'
2026-03-14 09:56:26.758773 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.758777 | controller |     - '        '
2026-03-14 09:56:26.758782 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.758786 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.758790 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.758801 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.758805 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.758809 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.758814 | controller |       -- \'
2026-03-14 09:56:26.758818 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.758831 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.758836 | controller |     - '    logger.go:42: 09:45:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.758840 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.758844 | controller |     - '    logger.go:42: 09:45:27 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.758849 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.758853 | controller |       found'
2026-03-14 09:56:26.758857 | controller |     - '    logger.go:42: 09:45:28 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.758861 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.758866 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.758870 | controller |     - '        '
2026-03-14 09:56:26.758874 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.758878 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.758883 | controller |     - '        '
2026-03-14 09:56:26.758887 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.758891 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.758900 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.758905 | controller |     - '          exit 1'
2026-03-14 09:56:26.758909 | controller |     - '        fi'
2026-03-14 09:56:26.758913 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.758918 | controller |     - '        '
2026-03-14 09:56:26.758922 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.758928 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.758932 | controller |       | base64 -d)'
2026-03-14 09:56:26.758937 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.758943 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.758952 | controller |     - '          exit 1'
2026-03-14 09:56:26.758956 | controller |     - '        fi'
2026-03-14 09:56:26.758961 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.758965 | controller |     - '        '
2026-03-14 09:56:26.758969 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.758973 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.758978 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.758982 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.758988 | controller |       --timeout=300s'
2026-03-14 09:56:26.758992 | controller |     - '        '
2026-03-14 09:56:26.758996 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.759001 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.759005 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.759009 | controller |     - '        '
2026-03-14 09:56:26.759031 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.759037 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.759043 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.759048 | controller |     - '        '
2026-03-14 09:56:26.759054 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.759059 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.759065 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.759070 | controller |       available"'
2026-03-14 09:56:26.759076 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.759081 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.759086 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.759091 | controller |     - '          exit 1'
2026-03-14 09:56:26.759096 | controller |     - '        fi'
2026-03-14 09:56:26.759101 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.759106 | controller |     - '        '
2026-03-14 09:56:26.759111 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.759116 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.759121 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.759126 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.759131 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.759137 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.759142 | controller |       -- \'
2026-03-14 09:56:26.759256 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.759271 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.759278 | controller |     - '    logger.go:42: 09:45:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.759287 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.759293 | controller |     - '    logger.go:42: 09:45:28 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.759298 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.759304 | controller |       found'
2026-03-14 09:56:26.759309 | controller |     - '    logger.go:42: 09:45:29 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.759314 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.759320 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.759325 | controller |     - '        '
2026-03-14 09:56:26.759331 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.759336 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.759342 | controller |     - '        '
2026-03-14 09:56:26.759347 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.759353 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.759358 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.759362 | controller |     - '          exit 1'
2026-03-14 09:56:26.759367 | controller |     - '        fi'
2026-03-14 09:56:26.759377 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.759382 | controller |     - '        '
2026-03-14 09:56:26.759390 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.759394 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.759399 | controller |       | base64 -d)'
2026-03-14 09:56:26.759403 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.759407 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.759411 | controller |     - '          exit 1'
2026-03-14 09:56:26.759416 | controller |     - '        fi'
2026-03-14 09:56:26.759420 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.759424 | controller |     - '        '
2026-03-14 09:56:26.759429 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.759433 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.759437 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.759441 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.759445 | controller |       --timeout=300s'
2026-03-14 09:56:26.759450 | controller |     - '        '
2026-03-14 09:56:26.759454 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.759461 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.759465 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.759471 | controller |     - '        '
2026-03-14 09:56:26.759475 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.759479 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.759484 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.759488 | controller |     - '        '
2026-03-14 09:56:26.759492 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.759496 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.759501 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.759506 | controller |       available"'
2026-03-14 09:56:26.759512 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.759521 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.759528 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.759533 | controller |     - '          exit 1'
2026-03-14 09:56:26.759538 | controller |     - '        fi'
2026-03-14 09:56:26.759542 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.759546 | controller |     - '        '
2026-03-14 09:56:26.759551 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.759555 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.759559 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.759563 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.759568 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.759572 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.759576 | controller |       -- \'
2026-03-14 09:56:26.759580 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.759585 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.759609 | controller |     - '    logger.go:42: 09:45:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.759616 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.759621 | controller |     - '    logger.go:42: 09:45:30 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.759625 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.759629 | controller |       found'
2026-03-14 09:56:26.759633 | controller |     - '    logger.go:42: 09:45:31 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.759638 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.759642 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.759646 | controller |     - '        '
2026-03-14 09:56:26.759651 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.759655 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.759659 | controller |     - '        '
2026-03-14 09:56:26.759663 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.759667 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.759672 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.759676 | controller |     - '          exit 1'
2026-03-14 09:56:26.759680 | controller |     - '        fi'
2026-03-14 09:56:26.759685 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.759689 | controller |     - '        '
2026-03-14 09:56:26.759693 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.759697 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.759702 | controller |       | base64 -d)'
2026-03-14 09:56:26.759706 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.759710 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.759714 | controller |     - '          exit 1'
2026-03-14 09:56:26.759719 | controller |     - '        fi'
2026-03-14 09:56:26.759723 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.759727 | controller |     - '        '
2026-03-14 09:56:26.759731 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.759735 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.759742 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.759746 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.759750 | controller |       --timeout=300s'
2026-03-14 09:56:26.759755 | controller |     - '        '
2026-03-14 09:56:26.759759 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.759763 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.759767 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.759772 | controller |     - '        '
2026-03-14 09:56:26.759776 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.759780 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.759784 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.759789 | controller |     - '        '
2026-03-14 09:56:26.759793 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.759802 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.759806 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.759811 | controller |       available"'
2026-03-14 09:56:26.759815 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.759819 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.759823 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.759827 | controller |     - '          exit 1'
2026-03-14 09:56:26.759832 | controller |     - '        fi'
2026-03-14 09:56:26.759836 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.759840 | controller |     - '        '
2026-03-14 09:56:26.759845 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.759849 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.759853 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.759859 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.759863 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.759868 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.759872 | controller |       -- \'
2026-03-14 09:56:26.759876 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.759880 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.759891 | controller |     - '    logger.go:42: 09:45:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.759895 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.759900 | controller |     - '    logger.go:42: 09:45:31 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.759904 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.759908 | controller |       found'
2026-03-14 09:56:26.759912 | controller |     - '    logger.go:42: 09:45:32 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.759917 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.759921 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.759925 | controller |     - '        '
2026-03-14 09:56:26.759936 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.759941 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.759945 | controller |     - '        '
2026-03-14 09:56:26.759949 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.759960 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.759966 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.759975 | controller |     - '          exit 1'
2026-03-14 09:56:26.759981 | controller |     - '        fi'
2026-03-14 09:56:26.759985 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.759990 | controller |     - '        '
2026-03-14 09:56:26.759998 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.760006 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.760010 | controller |       | base64 -d)'
2026-03-14 09:56:26.760015 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.760019 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.760024 | controller |     - '          exit 1'
2026-03-14 09:56:26.760032 | controller |     - '        fi'
2026-03-14 09:56:26.760037 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.760042 | controller |     - '        '
2026-03-14 09:56:26.760046 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.760051 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.760055 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.760060 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.760065 | controller |       --timeout=300s'
2026-03-14 09:56:26.760075 | controller |     - '        '
2026-03-14 09:56:26.760082 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.760086 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.760090 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.760095 | controller |     - '        '
2026-03-14 09:56:26.760099 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.760103 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.760107 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.760112 | controller |     - '        '
2026-03-14 09:56:26.760116 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.760120 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.760124 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.760129 | controller |       available"'
2026-03-14 09:56:26.760133 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.760137 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.760141 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.760145 | controller |     - '          exit 1'
2026-03-14 09:56:26.760150 | controller |     - '        fi'
2026-03-14 09:56:26.760154 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.760158 | controller |     - '        '
2026-03-14 09:56:26.760162 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.760169 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.760173 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.760177 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.760181 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.760186 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.760190 | controller |       -- \'
2026-03-14 09:56:26.760194 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.760198 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.760203 | controller |     - '    logger.go:42: 09:45:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.760213 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.760218 | controller |     - '    logger.go:42: 09:45:32 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.760222 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.760227 | controller |       found'
2026-03-14 09:56:26.760231 | controller |     - '    logger.go:42: 09:45:33 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.760235 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.760239 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.760244 | controller |     - '        '
2026-03-14 09:56:26.760248 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.760252 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.760256 | controller |     - '        '
2026-03-14 09:56:26.760260 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.760265 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.760269 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.760273 | controller |     - '          exit 1'
2026-03-14 09:56:26.760277 | controller |     - '        fi'
2026-03-14 09:56:26.760282 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.760286 | controller |     - '        '
2026-03-14 09:56:26.760290 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.760294 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.760299 | controller |       | base64 -d)'
2026-03-14 09:56:26.760303 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.760307 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.760311 | controller |     - '          exit 1'
2026-03-14 09:56:26.760316 | controller |     - '        fi'
2026-03-14 09:56:26.760320 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.760325 | controller |     - '        '
2026-03-14 09:56:26.760331 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.760337 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.760342 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.760348 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.760353 | controller |       --timeout=300s'
2026-03-14 09:56:26.760361 | controller |     - '        '
2026-03-14 09:56:26.760368 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.760374 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.760384 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.760390 | controller |     - '        '
2026-03-14 09:56:26.760395 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.760400 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.760404 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.760408 | controller |     - '        '
2026-03-14 09:56:26.760413 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.760417 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.760421 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.760428 | controller |       available"'
2026-03-14 09:56:26.760433 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.760437 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.760443 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.760447 | controller |     - '          exit 1'
2026-03-14 09:56:26.760451 | controller |     - '        fi'
2026-03-14 09:56:26.760455 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.760460 | controller |     - '        '
2026-03-14 09:56:26.760464 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.760468 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.760472 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.760477 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.760481 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.760485 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.760489 | controller |       -- \'
2026-03-14 09:56:26.760494 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.760498 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.760502 | controller |     - '    logger.go:42: 09:45:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.760506 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.760518 | controller |     - '    logger.go:42: 09:45:33 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.760524 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.760530 | controller |       found'
2026-03-14 09:56:26.760536 | controller |     - '    logger.go:42: 09:45:34 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.760542 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.760548 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.760553 | controller |     - '        '
2026-03-14 09:56:26.760557 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.760561 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.760566 | controller |     - '        '
2026-03-14 09:56:26.760570 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.760574 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.760587 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.760610 | controller |     - '          exit 1'
2026-03-14 09:56:26.760618 | controller |     - '        fi'
2026-03-14 09:56:26.760622 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.760626 | controller |     - '        '
2026-03-14 09:56:26.760631 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.760635 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.760639 | controller |       | base64 -d)'
2026-03-14 09:56:26.760643 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.760648 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.760652 | controller |     - '          exit 1'
2026-03-14 09:56:26.760656 | controller |     - '        fi'
2026-03-14 09:56:26.760660 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.760665 | controller |     - '        '
2026-03-14 09:56:26.760669 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.760673 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.760677 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.760682 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.760686 | controller |       --timeout=300s'
2026-03-14 09:56:26.760690 | controller |     - '        '
2026-03-14 09:56:26.760695 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.760699 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.760703 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.760707 | controller |     - '        '
2026-03-14 09:56:26.760712 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.760716 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.760720 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.760725 | controller |     - '        '
2026-03-14 09:56:26.760729 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.760733 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.760737 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.760742 | controller |       available"'
2026-03-14 09:56:26.760746 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.760750 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.760754 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.760759 | controller |     - '          exit 1'
2026-03-14 09:56:26.760764 | controller |     - '        fi'
2026-03-14 09:56:26.760770 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.760776 | controller |     - '        '
2026-03-14 09:56:26.760782 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.760788 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.760794 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.760799 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.760804 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.760808 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.760815 | controller |       -- \'
2026-03-14 09:56:26.760819 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.760823 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.760828 | controller |     - '    logger.go:42: 09:45:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.760832 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.760837 | controller |     - '    logger.go:42: 09:45:34 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.760849 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.760854 | controller |       found'
2026-03-14 09:56:26.760858 | controller |     - '    logger.go:42: 09:45:35 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.760863 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.760867 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.760872 | controller |     - '        '
2026-03-14 09:56:26.760876 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.760880 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.760885 | controller |     - '        '
2026-03-14 09:56:26.760889 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.760893 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.760897 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.760902 | controller |     - '          exit 1'
2026-03-14 09:56:26.760906 | controller |     - '        fi'
2026-03-14 09:56:26.760910 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.760917 | controller |     - '        '
2026-03-14 09:56:26.760921 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.760926 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.760933 | controller |       | base64 -d)'
2026-03-14 09:56:26.760937 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.760941 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.760945 | controller |     - '          exit 1'
2026-03-14 09:56:26.760950 | controller |     - '        fi'
2026-03-14 09:56:26.760954 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.760958 | controller |     - '        '
2026-03-14 09:56:26.760962 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.760967 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.760971 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.760975 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.760980 | controller |       --timeout=300s'
2026-03-14 09:56:26.760984 | controller |     - '        '
2026-03-14 09:56:26.760988 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.760992 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.760997 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.761003 | controller |     - '        '
2026-03-14 09:56:26.761007 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.761011 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.761016 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.761022 | controller |     - '        '
2026-03-14 09:56:26.761026 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.761030 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.761035 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.761039 | controller |       available"'
2026-03-14 09:56:26.761043 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.761048 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.761052 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.761056 | controller |     - '          exit 1'
2026-03-14 09:56:26.761060 | controller |     - '        fi'
2026-03-14 09:56:26.761065 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.761072 | controller |     - '        '
2026-03-14 09:56:26.761076 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.761081 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.761085 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.761089 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.761094 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.761098 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.761102 | controller |       -- \'
2026-03-14 09:56:26.761106 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.761111 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.761115 | controller |     - '    logger.go:42: 09:45:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.761119 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.761124 | controller |     - '    logger.go:42: 09:45:35 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.761134 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.761139 | controller |       found'
2026-03-14 09:56:26.761143 | controller |     - '    logger.go:42: 09:45:36 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.761147 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.761152 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.761156 | controller |     - '        '
2026-03-14 09:56:26.761160 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.761165 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.761169 | controller |     - '        '
2026-03-14 09:56:26.761173 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.761177 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.761182 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.761186 | controller |     - '          exit 1'
2026-03-14 09:56:26.761190 | controller |     - '        fi'
2026-03-14 09:56:26.761195 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.761199 | controller |     - '        '
2026-03-14 09:56:26.761205 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.761209 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.761214 | controller |       | base64 -d)'
2026-03-14 09:56:26.761218 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.761224 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.761228 | controller |     - '          exit 1'
2026-03-14 09:56:26.761232 | controller |     - '        fi'
2026-03-14 09:56:26.761237 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.761241 | controller |     - '        '
2026-03-14 09:56:26.761245 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.761249 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.761254 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.761258 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.761262 | controller |       --timeout=300s'
2026-03-14 09:56:26.761266 | controller |     - '        '
2026-03-14 09:56:26.761271 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.761275 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.761279 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.761283 | controller |     - '        '
2026-03-14 09:56:26.761288 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.761292 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.761296 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.761300 | controller |     - '        '
2026-03-14 09:56:26.761305 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.761309 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.761313 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.761318 | controller |       available"'
2026-03-14 09:56:26.761322 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.761326 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.761330 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.761335 | controller |     - '          exit 1'
2026-03-14 09:56:26.761339 | controller |     - '        fi'
2026-03-14 09:56:26.761343 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.761347 | controller |     - '        '
2026-03-14 09:56:26.761352 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.761356 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.761360 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.761364 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.761369 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.761373 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.761377 | controller |       -- \'
2026-03-14 09:56:26.761381 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.761386 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.761390 | controller |     - '    logger.go:42: 09:45:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.761394 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.761400 | controller |     - '    logger.go:42: 09:45:37 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.761404 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.761408 | controller |       found'
2026-03-14 09:56:26.761419 | controller |     - '    logger.go:42: 09:45:38 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.761423 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.761427 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.761432 | controller |     - '        '
2026-03-14 09:56:26.761436 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.761440 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.761444 | controller |     - '        '
2026-03-14 09:56:26.761449 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.761453 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.761457 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.761461 | controller |     - '          exit 1'
2026-03-14 09:56:26.761466 | controller |     - '        fi'
2026-03-14 09:56:26.761470 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.761474 | controller |     - '        '
2026-03-14 09:56:26.761478 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.761483 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.761487 | controller |       | base64 -d)'
2026-03-14 09:56:26.761491 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.761495 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.761500 | controller |     - '          exit 1'
2026-03-14 09:56:26.761504 | controller |     - '        fi'
2026-03-14 09:56:26.761508 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.761512 | controller |     - '        '
2026-03-14 09:56:26.761517 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.761521 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.761525 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.761529 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.761534 | controller |       --timeout=300s'
2026-03-14 09:56:26.761539 | controller |     - '        '
2026-03-14 09:56:26.761545 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.761551 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.761557 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.761564 | controller |     - '        '
2026-03-14 09:56:26.761569 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.761574 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.761578 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.761582 | controller |     - '        '
2026-03-14 09:56:26.761586 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.761591 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.761610 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.761614 | controller |       available"'
2026-03-14 09:56:26.761619 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.761623 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.761630 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.761634 | controller |     - '          exit 1'
2026-03-14 09:56:26.761638 | controller |     - '        fi'
2026-03-14 09:56:26.761642 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.761647 | controller |     - '        '
2026-03-14 09:56:26.761651 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.761665 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.761670 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.761674 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.761678 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.761682 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.761687 | controller |       -- \'
2026-03-14 09:56:26.761691 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.761695 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.761699 | controller |     - '    logger.go:42: 09:45:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.761704 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.761708 | controller |     - '    logger.go:42: 09:45:38 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.761712 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.761717 | controller |       found'
2026-03-14 09:56:26.761729 | controller |     - '    logger.go:42: 09:45:39 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.761733 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.761738 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.761742 | controller |     - '        '
2026-03-14 09:56:26.761746 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.761751 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.761755 | controller |     - '        '
2026-03-14 09:56:26.761759 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.761763 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.761767 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.761772 | controller |     - '          exit 1'
2026-03-14 09:56:26.761776 | controller |     - '        fi'
2026-03-14 09:56:26.761780 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.761785 | controller |     - '        '
2026-03-14 09:56:26.761789 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.761795 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.761799 | controller |       | base64 -d)'
2026-03-14 09:56:26.761803 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.761808 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.761812 | controller |     - '          exit 1'
2026-03-14 09:56:26.761816 | controller |     - '        fi'
2026-03-14 09:56:26.761820 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.761825 | controller |     - '        '
2026-03-14 09:56:26.761829 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.761833 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.761837 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.761844 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.761849 | controller |       --timeout=300s'
2026-03-14 09:56:26.761853 | controller |     - '        '
2026-03-14 09:56:26.761857 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.761862 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.761866 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.761870 | controller |     - '        '
2026-03-14 09:56:26.761874 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.761879 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.761883 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.761887 | controller |     - '        '
2026-03-14 09:56:26.761891 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.761896 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.761900 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.761904 | controller |       available"'
2026-03-14 09:56:26.761908 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.761913 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.761917 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.761921 | controller |     - '          exit 1'
2026-03-14 09:56:26.761925 | controller |     - '        fi'
2026-03-14 09:56:26.761930 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.761934 | controller |     - '        '
2026-03-14 09:56:26.761940 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.761944 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.761949 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.761953 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.761957 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.761963 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.761967 | controller |       -- \'
2026-03-14 09:56:26.761972 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.761976 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.761980 | controller |     - '    logger.go:42: 09:45:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.761984 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.761989 | controller |     - '    logger.go:42: 09:45:39 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.761993 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.761997 | controller |       found'
2026-03-14 09:56:26.762002 | controller |     - '    logger.go:42: 09:45:40 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.762006 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.762016 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.762021 | controller |     - '        '
2026-03-14 09:56:26.762025 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.762031 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.762035 | controller |     - '        '
2026-03-14 09:56:26.762040 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.762044 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.762048 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.762052 | controller |     - '          exit 1'
2026-03-14 09:56:26.762057 | controller |     - '        fi'
2026-03-14 09:56:26.762061 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.762065 | controller |     - '        '
2026-03-14 09:56:26.762069 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.762074 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.762078 | controller |       | base64 -d)'
2026-03-14 09:56:26.762082 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.762086 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.762091 | controller |     - '          exit 1'
2026-03-14 09:56:26.762095 | controller |     - '        fi'
2026-03-14 09:56:26.762099 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.762103 | controller |     - '        '
2026-03-14 09:56:26.762108 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.762112 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.762116 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.762120 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.762125 | controller |       --timeout=300s'
2026-03-14 09:56:26.762129 | controller |     - '        '
2026-03-14 09:56:26.762133 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.762137 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.762142 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.762146 | controller |     - '        '
2026-03-14 09:56:26.762150 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.762154 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.762159 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.762163 | controller |     - '        '
2026-03-14 09:56:26.762167 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.762171 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.762176 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.762180 | controller |       available"'
2026-03-14 09:56:26.762184 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.762188 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.762193 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.762197 | controller |     - '          exit 1'
2026-03-14 09:56:26.762201 | controller |     - '        fi'
2026-03-14 09:56:26.762206 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.762210 | controller |     - '        '
2026-03-14 09:56:26.762216 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.762221 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.762226 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.762231 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.762237 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.762241 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.762247 | controller |       -- \'
2026-03-14 09:56:26.762251 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.762256 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.762260 | controller |     - '    logger.go:42: 09:45:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.762264 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.762268 | controller |     - '    logger.go:42: 09:45:40 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.762273 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.762277 | controller |       found'
2026-03-14 09:56:26.762281 | controller |     - '    logger.go:42: 09:45:41 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.762286 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.762290 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.762294 | controller |     - '        '
2026-03-14 09:56:26.762305 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.762309 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.762314 | controller |     - '        '
2026-03-14 09:56:26.762318 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.762322 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.762327 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.762331 | controller |     - '          exit 1'
2026-03-14 09:56:26.762335 | controller |     - '        fi'
2026-03-14 09:56:26.762339 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.762344 | controller |     - '        '
2026-03-14 09:56:26.762348 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.762352 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.762357 | controller |       | base64 -d)'
2026-03-14 09:56:26.762361 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.762365 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.762370 | controller |     - '          exit 1'
2026-03-14 09:56:26.762374 | controller |     - '        fi'
2026-03-14 09:56:26.762378 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.762382 | controller |     - '        '
2026-03-14 09:56:26.762387 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.762391 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.762395 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.762399 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.762404 | controller |       --timeout=300s'
2026-03-14 09:56:26.762408 | controller |     - '        '
2026-03-14 09:56:26.762412 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.762416 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.762421 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.762428 | controller |     - '        '
2026-03-14 09:56:26.762432 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.762437 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.762441 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.762445 | controller |     - '        '
2026-03-14 09:56:26.762450 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.762454 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.762458 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.762462 | controller |       available"'
2026-03-14 09:56:26.762467 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.762471 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.762475 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.762480 | controller |     - '          exit 1'
2026-03-14 09:56:26.762484 | controller |     - '        fi'
2026-03-14 09:56:26.762488 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.762492 | controller |     - '        '
2026-03-14 09:56:26.762497 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.762501 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.762505 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.762509 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.762514 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.762518 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.762522 | controller |       -- \'
2026-03-14 09:56:26.762527 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.762531 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.762535 | controller |     - '    logger.go:42: 09:45:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.762539 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.762544 | controller |     - '    logger.go:42: 09:45:41 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.762548 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.762552 | controller |       found'
2026-03-14 09:56:26.762558 | controller |     - '    logger.go:42: 09:45:42 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.762563 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.762570 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.762576 | controller |     - '        '
2026-03-14 09:56:26.762589 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.762606 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.762613 | controller |     - '        '
2026-03-14 09:56:26.762617 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.762621 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.762626 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.762630 | controller |     - '          exit 1'
2026-03-14 09:56:26.762634 | controller |     - '        fi'
2026-03-14 09:56:26.762641 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.762645 | controller |     - '        '
2026-03-14 09:56:26.762650 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.762654 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.762658 | controller |       | base64 -d)'
2026-03-14 09:56:26.762663 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.762667 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.762671 | controller |     - '          exit 1'
2026-03-14 09:56:26.762675 | controller |     - '        fi'
2026-03-14 09:56:26.762680 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.762684 | controller |     - '        '
2026-03-14 09:56:26.762688 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.762692 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.762697 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.762701 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.762705 | controller |       --timeout=300s'
2026-03-14 09:56:26.762709 | controller |     - '        '
2026-03-14 09:56:26.762714 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.762718 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.762722 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.762726 | controller |     - '        '
2026-03-14 09:56:26.762731 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.762735 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.762739 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.762744 | controller |     - '        '
2026-03-14 09:56:26.762748 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.762752 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.762756 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.762761 | controller |       available"'
2026-03-14 09:56:26.762765 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.762769 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.762773 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.762778 | controller |     - '          exit 1'
2026-03-14 09:56:26.762782 | controller |     - '        fi'
2026-03-14 09:56:26.762786 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.762790 | controller |     - '        '
2026-03-14 09:56:26.762795 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.762799 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.762803 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.762807 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.762812 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.762816 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.762822 | controller |       -- \'
2026-03-14 09:56:26.762826 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.762831 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.762837 | controller |     - '    logger.go:42: 09:45:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.762842 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.762846 | controller |     - '    logger.go:42: 09:45:42 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.762850 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.762854 | controller |       found'
2026-03-14 09:56:26.762859 | controller |     - '    logger.go:42: 09:45:43 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.762863 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.762867 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.762871 | controller |     - '        '
2026-03-14 09:56:26.762876 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.762887 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.762891 | controller |     - '        '
2026-03-14 09:56:26.762896 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.762900 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.762904 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.762909 | controller |     - '          exit 1'
2026-03-14 09:56:26.762913 | controller |     - '        fi'
2026-03-14 09:56:26.762917 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.762921 | controller |     - '        '
2026-03-14 09:56:26.762925 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.762930 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.762934 | controller |       | base64 -d)'
2026-03-14 09:56:26.762938 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.762942 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.762947 | controller |     - '          exit 1'
2026-03-14 09:56:26.762951 | controller |     - '        fi'
2026-03-14 09:56:26.762955 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.762959 | controller |     - '        '
2026-03-14 09:56:26.762963 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.762968 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.762972 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.762976 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.762980 | controller |       --timeout=300s'
2026-03-14 09:56:26.762985 | controller |     - '        '
2026-03-14 09:56:26.762989 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.762993 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.762997 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.763002 | controller |     - '        '
2026-03-14 09:56:26.763006 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.763010 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.763031 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.763039 | controller |     - '        '
2026-03-14 09:56:26.763044 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.763048 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.763052 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.763056 | controller |       available"'
2026-03-14 09:56:26.763061 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.763065 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.763071 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.763075 | controller |     - '          exit 1'
2026-03-14 09:56:26.763080 | controller |     - '        fi'
2026-03-14 09:56:26.763084 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.763088 | controller |     - '        '
2026-03-14 09:56:26.763092 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.763097 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.763101 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.763105 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.763109 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.763113 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.763118 | controller |       -- \'
2026-03-14 09:56:26.763122 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.763126 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.763130 | controller |     - '    logger.go:42: 09:45:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.763135 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.763139 | controller |     - '    logger.go:42: 09:45:44 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.763143 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.763147 | controller |       found'
2026-03-14 09:56:26.763152 | controller |     - '    logger.go:42: 09:45:45 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.763156 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.763160 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.763164 | controller |     - '        '
2026-03-14 09:56:26.763169 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.763180 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.763184 | controller |     - '        '
2026-03-14 09:56:26.763189 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.763193 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.763197 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.763201 | controller |     - '          exit 1'
2026-03-14 09:56:26.763206 | controller |     - '        fi'
2026-03-14 09:56:26.763210 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.763214 | controller |     - '        '
2026-03-14 09:56:26.763218 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.763223 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.763227 | controller |       | base64 -d)'
2026-03-14 09:56:26.763231 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.763236 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.763241 | controller |     - '          exit 1'
2026-03-14 09:56:26.763246 | controller |     - '        fi'
2026-03-14 09:56:26.763250 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.763254 | controller |     - '        '
2026-03-14 09:56:26.763259 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.763263 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.763267 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.763271 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.763276 | controller |       --timeout=300s'
2026-03-14 09:56:26.763280 | controller |     - '        '
2026-03-14 09:56:26.763284 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.763289 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.763294 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.763299 | controller |     - '        '
2026-03-14 09:56:26.763303 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.763307 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.763312 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.763316 | controller |     - '        '
2026-03-14 09:56:26.763320 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.763324 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.763329 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.763333 | controller |       available"'
2026-03-14 09:56:26.763337 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.763341 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.763346 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.763350 | controller |     - '          exit 1'
2026-03-14 09:56:26.763354 | controller |     - '        fi'
2026-03-14 09:56:26.763358 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.763363 | controller |     - '        '
2026-03-14 09:56:26.763367 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.763371 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.763375 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.763380 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.763384 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.763388 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.763392 | controller |       -- \'
2026-03-14 09:56:26.763397 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.763401 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.763405 | controller |     - '    logger.go:42: 09:45:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.763409 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.763414 | controller |     - '    logger.go:42: 09:45:45 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.763421 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.763425 | controller |       found'
2026-03-14 09:56:26.763430 | controller |     - '    logger.go:42: 09:45:46 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.763434 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.763439 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.763445 | controller |     - '        '
2026-03-14 09:56:26.763450 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.763457 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.763463 | controller |     - '        '
2026-03-14 09:56:26.763475 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.763481 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.763487 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.763492 | controller |     - '          exit 1'
2026-03-14 09:56:26.763496 | controller |     - '        fi'
2026-03-14 09:56:26.763500 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.763504 | controller |     - '        '
2026-03-14 09:56:26.763509 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.763513 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.763517 | controller |       | base64 -d)'
2026-03-14 09:56:26.763521 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.763526 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.763530 | controller |     - '          exit 1'
2026-03-14 09:56:26.763534 | controller |     - '        fi'
2026-03-14 09:56:26.763539 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.763543 | controller |     - '        '
2026-03-14 09:56:26.763547 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.763551 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.763556 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.763560 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.763564 | controller |       --timeout=300s'
2026-03-14 09:56:26.763568 | controller |     - '        '
2026-03-14 09:56:26.763574 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.763583 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.763589 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.763611 | controller |     - '        '
2026-03-14 09:56:26.763620 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.763626 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.763632 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.763636 | controller |     - '        '
2026-03-14 09:56:26.763642 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.763648 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.763654 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.763658 | controller |       available"'
2026-03-14 09:56:26.763664 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.763670 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.763678 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.763683 | controller |     - '          exit 1'
2026-03-14 09:56:26.763689 | controller |     - '        fi'
2026-03-14 09:56:26.763695 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.763700 | controller |     - '        '
2026-03-14 09:56:26.763704 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.763708 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.763713 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.763717 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.763721 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.763726 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.763730 | controller |       -- \'
2026-03-14 09:56:26.763734 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.763738 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.763743 | controller |     - '    logger.go:42: 09:45:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.763747 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.763751 | controller |     - '    logger.go:42: 09:45:46 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.763755 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.763760 | controller |       found'
2026-03-14 09:56:26.763764 | controller |     - '    logger.go:42: 09:45:47 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.763768 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.763772 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.763777 | controller |     - '        '
2026-03-14 09:56:26.763781 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.763785 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.763789 | controller |     - '        '
2026-03-14 09:56:26.763801 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.763806 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.763810 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.763815 | controller |     - '          exit 1'
2026-03-14 09:56:26.763819 | controller |     - '        fi'
2026-03-14 09:56:26.763823 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.763828 | controller |     - '        '
2026-03-14 09:56:26.763832 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.763836 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.763841 | controller |       | base64 -d)'
2026-03-14 09:56:26.763847 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.763853 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.763857 | controller |     - '          exit 1'
2026-03-14 09:56:26.763863 | controller |     - '        fi'
2026-03-14 09:56:26.763869 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.763874 | controller |     - '        '
2026-03-14 09:56:26.763879 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.763885 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.763891 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.763898 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.763902 | controller |       --timeout=300s'
2026-03-14 09:56:26.763908 | controller |     - '        '
2026-03-14 09:56:26.763914 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.763919 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.763924 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.763929 | controller |     - '        '
2026-03-14 09:56:26.763935 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.763941 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.763946 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.763952 | controller |     - '        '
2026-03-14 09:56:26.763958 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.763963 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.763969 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.763975 | controller |       available"'
2026-03-14 09:56:26.763980 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.763985 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.763989 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.763994 | controller |     - '          exit 1'
2026-03-14 09:56:26.764000 | controller |     - '        fi'
2026-03-14 09:56:26.764006 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.764011 | controller |     - '        '
2026-03-14 09:56:26.764015 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.764019 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.764023 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.764028 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.764032 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.764036 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.764041 | controller |       -- \'
2026-03-14 09:56:26.764045 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.764052 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.764057 | controller |     - '    logger.go:42: 09:45:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.764061 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.764066 | controller |     - '    logger.go:42: 09:45:47 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.764070 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.764077 | controller |       found'
2026-03-14 09:56:26.764082 | controller |     - '    logger.go:42: 09:45:48 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.764086 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.764090 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.764094 | controller |     - '        '
2026-03-14 09:56:26.764099 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.764105 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.764109 | controller |     - '        '
2026-03-14 09:56:26.764113 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.764126 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.764133 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.764139 | controller |     - '          exit 1'
2026-03-14 09:56:26.764144 | controller |     - '        fi'
2026-03-14 09:56:26.764151 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.764157 | controller |     - '        '
2026-03-14 09:56:26.764161 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.764167 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.764173 | controller |       | base64 -d)'
2026-03-14 09:56:26.764179 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.764183 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.764188 | controller |     - '          exit 1'
2026-03-14 09:56:26.764194 | controller |     - '        fi'
2026-03-14 09:56:26.764200 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.764204 | controller |     - '        '
2026-03-14 09:56:26.764208 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.764212 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.764217 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.764221 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.764225 | controller |       --timeout=300s'
2026-03-14 09:56:26.764229 | controller |     - '        '
2026-03-14 09:56:26.764234 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.764238 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.764242 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.764247 | controller |     - '        '
2026-03-14 09:56:26.764251 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.764255 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.764259 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.764264 | controller |     - '        '
2026-03-14 09:56:26.764268 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.764272 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.764276 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.764281 | controller |       available"'
2026-03-14 09:56:26.764285 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.764289 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.764293 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.764298 | controller |     - '          exit 1'
2026-03-14 09:56:26.764302 | controller |     - '        fi'
2026-03-14 09:56:26.764306 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.764310 | controller |     - '        '
2026-03-14 09:56:26.764315 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.764319 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.764327 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.764333 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.764338 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.764343 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.764349 | controller |       -- \'
2026-03-14 09:56:26.764355 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.764360 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.764365 | controller |     - '    logger.go:42: 09:45:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.764371 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.764377 | controller |     - '    logger.go:42: 09:45:48 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.764381 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.764386 | controller |       found'
2026-03-14 09:56:26.764392 | controller |     - '    logger.go:42: 09:45:49 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.764398 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.764405 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.764410 | controller |     - '        '
2026-03-14 09:56:26.764414 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.764419 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.764423 | controller |     - '        '
2026-03-14 09:56:26.764427 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.764431 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.764443 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.764448 | controller |     - '          exit 1'
2026-03-14 09:56:26.764452 | controller |     - '        fi'
2026-03-14 09:56:26.764457 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.764461 | controller |     - '        '
2026-03-14 09:56:26.764465 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.764471 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.764476 | controller |       | base64 -d)'
2026-03-14 09:56:26.764480 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.764484 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.764489 | controller |     - '          exit 1'
2026-03-14 09:56:26.764495 | controller |     - '        fi'
2026-03-14 09:56:26.764499 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.764503 | controller |     - '        '
2026-03-14 09:56:26.764507 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.764512 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.764516 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.764520 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.764525 | controller |       --timeout=300s'
2026-03-14 09:56:26.764531 | controller |     - '        '
2026-03-14 09:56:26.764537 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.764543 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.764549 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.764557 | controller |     - '        '
2026-03-14 09:56:26.764562 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.764568 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.764574 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.764579 | controller |     - '        '
2026-03-14 09:56:26.764584 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.764590 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.764708 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.764715 | controller |       available"'
2026-03-14 09:56:26.764719 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.764724 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.764728 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.764733 | controller |     - '          exit 1'
2026-03-14 09:56:26.764737 | controller |     - '        fi'
2026-03-14 09:56:26.764741 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.764746 | controller |     - '        '
2026-03-14 09:56:26.764750 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.764754 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.764759 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.764763 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.764767 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.764772 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.764776 | controller |       -- \'
2026-03-14 09:56:26.764780 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.764784 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.764789 | controller |     - '    logger.go:42: 09:45:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.764793 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.764797 | controller |     - '    logger.go:42: 09:45:50 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.764802 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.764806 | controller |       found'
2026-03-14 09:56:26.764810 | controller |     - '    logger.go:42: 09:45:51 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.764818 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.764822 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.764827 | controller |     - '        '
2026-03-14 09:56:26.764831 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.764835 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.764839 | controller |     - '        '
2026-03-14 09:56:26.764844 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.764848 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.764852 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.764865 | controller |     - '          exit 1'
2026-03-14 09:56:26.764870 | controller |     - '        fi'
2026-03-14 09:56:26.764875 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.764882 | controller |     - '        '
2026-03-14 09:56:26.764886 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.764890 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.764895 | controller |       | base64 -d)'
2026-03-14 09:56:26.764899 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.764903 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.764907 | controller |     - '          exit 1'
2026-03-14 09:56:26.764912 | controller |     - '        fi'
2026-03-14 09:56:26.764916 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.764920 | controller |     - '        '
2026-03-14 09:56:26.764925 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.764929 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.764933 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.764938 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.764942 | controller |       --timeout=300s'
2026-03-14 09:56:26.764946 | controller |     - '        '
2026-03-14 09:56:26.764950 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.764955 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.764959 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.764963 | controller |     - '        '
2026-03-14 09:56:26.764968 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.764972 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.764976 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.764982 | controller |     - '        '
2026-03-14 09:56:26.764986 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.764990 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.764995 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.764999 | controller |       available"'
2026-03-14 09:56:26.765003 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.765007 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.765012 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.765016 | controller |     - '          exit 1'
2026-03-14 09:56:26.765020 | controller |     - '        fi'
2026-03-14 09:56:26.765025 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.765029 | controller |     - '        '
2026-03-14 09:56:26.765033 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.765038 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.765042 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.765046 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.765051 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.765055 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.765059 | controller |       -- \'
2026-03-14 09:56:26.765064 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.765070 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.765074 | controller |     - '    logger.go:42: 09:45:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.765078 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.765083 | controller |     - '    logger.go:42: 09:45:51 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.765087 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.765091 | controller |       found'
2026-03-14 09:56:26.765095 | controller |     - '    logger.go:42: 09:45:52 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.765100 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.765104 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.765109 | controller |     - '        '
2026-03-14 09:56:26.765113 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.765117 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.765121 | controller |     - '        '
2026-03-14 09:56:26.765126 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.765130 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.765134 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.765139 | controller |     - '          exit 1'
2026-03-14 09:56:26.765143 | controller |     - '        fi'
2026-03-14 09:56:26.765166 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.765174 | controller |     - '        '
2026-03-14 09:56:26.765178 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.765182 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.765188 | controller |       | base64 -d)'
2026-03-14 09:56:26.765193 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.765197 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.765201 | controller |     - '          exit 1'
2026-03-14 09:56:26.765206 | controller |     - '        fi'
2026-03-14 09:56:26.765210 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.765214 | controller |     - '        '
2026-03-14 09:56:26.765219 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.765223 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.765227 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.765232 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.765236 | controller |       --timeout=300s'
2026-03-14 09:56:26.765240 | controller |     - '        '
2026-03-14 09:56:26.765244 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.765249 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.765253 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.765257 | controller |     - '        '
2026-03-14 09:56:26.765262 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.765266 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.765270 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.765276 | controller |     - '        '
2026-03-14 09:56:26.765281 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.765287 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.765291 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.765296 | controller |       available"'
2026-03-14 09:56:26.765300 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.765305 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.765310 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.765314 | controller |     - '          exit 1'
2026-03-14 09:56:26.765318 | controller |     - '        fi'
2026-03-14 09:56:26.765323 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.765327 | controller |     - '        '
2026-03-14 09:56:26.765333 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.765338 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.765342 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.765347 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.765351 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.765355 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.765360 | controller |       -- \'
2026-03-14 09:56:26.765364 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.765368 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.765373 | controller |     - '    logger.go:42: 09:45:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.765377 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.765381 | controller |     - '    logger.go:42: 09:45:52 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.765386 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.765390 | controller |       found'
2026-03-14 09:56:26.765394 | controller |     - '    logger.go:42: 09:45:53 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.765399 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.765403 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.765407 | controller |     - '        '
2026-03-14 09:56:26.765412 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.765419 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.765424 | controller |     - '        '
2026-03-14 09:56:26.765428 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.765432 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.765437 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.765441 | controller |     - '          exit 1'
2026-03-14 09:56:26.765446 | controller |     - '        fi'
2026-03-14 09:56:26.765450 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.765454 | controller |     - '        '
2026-03-14 09:56:26.765466 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.765471 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.765475 | controller |       | base64 -d)'
2026-03-14 09:56:26.765480 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.765484 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.765488 | controller |     - '          exit 1'
2026-03-14 09:56:26.765495 | controller |     - '        fi'
2026-03-14 09:56:26.765499 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.765503 | controller |     - '        '
2026-03-14 09:56:26.765508 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.765512 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.765516 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.765521 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.765525 | controller |       --timeout=300s'
2026-03-14 09:56:26.765529 | controller |     - '        '
2026-03-14 09:56:26.765534 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.765538 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.765542 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.765547 | controller |     - '        '
2026-03-14 09:56:26.765551 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.765555 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.765560 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.765564 | controller |     - '        '
2026-03-14 09:56:26.765568 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.765573 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.765577 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.765581 | controller |       available"'
2026-03-14 09:56:26.765585 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.765590 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.765608 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.765617 | controller |     - '          exit 1'
2026-03-14 09:56:26.765624 | controller |     - '        fi'
2026-03-14 09:56:26.765631 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.765635 | controller |     - '        '
2026-03-14 09:56:26.765639 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.765644 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.765648 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.765653 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.765657 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.765661 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.765666 | controller |       -- \'
2026-03-14 09:56:26.765670 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.765674 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.765679 | controller |     - '    logger.go:42: 09:45:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.765683 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.765687 | controller |     - '    logger.go:42: 09:45:53 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.765692 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.765698 | controller |       found'
2026-03-14 09:56:26.765702 | controller |     - '    logger.go:42: 09:45:54 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.765707 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.765711 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.765715 | controller |     - '        '
2026-03-14 09:56:26.765720 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.765724 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.765729 | controller |     - '        '
2026-03-14 09:56:26.765735 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.765739 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.765744 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.765748 | controller |     - '          exit 1'
2026-03-14 09:56:26.765752 | controller |     - '        fi'
2026-03-14 09:56:26.765757 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.765761 | controller |     - '        '
2026-03-14 09:56:26.765774 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.765778 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.765783 | controller |       | base64 -d)'
2026-03-14 09:56:26.765787 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.765791 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.765796 | controller |     - '          exit 1'
2026-03-14 09:56:26.765800 | controller |     - '        fi'
2026-03-14 09:56:26.765804 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.765809 | controller |     - '        '
2026-03-14 09:56:26.765813 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.765817 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.765822 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.765826 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.765830 | controller |       --timeout=300s'
2026-03-14 09:56:26.765834 | controller |     - '        '
2026-03-14 09:56:26.765839 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.765843 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.765847 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.765852 | controller |     - '        '
2026-03-14 09:56:26.765856 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.765860 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.765865 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.765869 | controller |     - '        '
2026-03-14 09:56:26.765873 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.765877 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.765882 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.765886 | controller |       available"'
2026-03-14 09:56:26.765890 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.765895 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.765901 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.765907 | controller |     - '          exit 1'
2026-03-14 09:56:26.765911 | controller |     - '        fi'
2026-03-14 09:56:26.765916 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.765920 | controller |     - '        '
2026-03-14 09:56:26.765924 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.765929 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.765933 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.765937 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.765942 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.765946 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.765950 | controller |       -- \'
2026-03-14 09:56:26.765955 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.765959 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.765963 | controller |     - '    logger.go:42: 09:45:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.765968 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.765972 | controller |     - '    logger.go:42: 09:45:54 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.765976 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.765981 | controller |       found'
2026-03-14 09:56:26.765985 | controller |     - '    logger.go:42: 09:45:55 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.765989 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.765993 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.765998 | controller |     - '        '
2026-03-14 09:56:26.766002 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.766006 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.766012 | controller |     - '        '
2026-03-14 09:56:26.766017 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.766021 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.766025 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.766030 | controller |     - '          exit 1'
2026-03-14 09:56:26.766034 | controller |     - '        fi'
2026-03-14 09:56:26.766038 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.766043 | controller |     - '        '
2026-03-14 09:56:26.766047 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.766057 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.766062 | controller |       | base64 -d)'
2026-03-14 09:56:26.766066 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.766071 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.766075 | controller |     - '          exit 1'
2026-03-14 09:56:26.766079 | controller |     - '        fi'
2026-03-14 09:56:26.766084 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.766088 | controller |     - '        '
2026-03-14 09:56:26.766092 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.766097 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.766101 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.766105 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.766111 | controller |       --timeout=300s'
2026-03-14 09:56:26.766116 | controller |     - '        '
2026-03-14 09:56:26.766120 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.766124 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.766129 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.766133 | controller |     - '        '
2026-03-14 09:56:26.766137 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.766142 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.766146 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.766150 | controller |     - '        '
2026-03-14 09:56:26.766155 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.766159 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.766163 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.766167 | controller |       available"'
2026-03-14 09:56:26.766172 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.766176 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.766180 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.766185 | controller |     - '          exit 1'
2026-03-14 09:56:26.766189 | controller |     - '        fi'
2026-03-14 09:56:26.766193 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.766198 | controller |     - '        '
2026-03-14 09:56:26.766202 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.766206 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.766210 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.766215 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.766219 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.766223 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.766228 | controller |       -- \'
2026-03-14 09:56:26.766232 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.766236 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.766241 | controller |     - '    logger.go:42: 09:45:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.766245 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.766249 | controller |     - '    logger.go:42: 09:45:55 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.766254 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.766258 | controller |       found'
2026-03-14 09:56:26.766262 | controller |     - '    logger.go:42: 09:45:57 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.766267 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.766271 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.766275 | controller |     - '        '
2026-03-14 09:56:26.766280 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.766284 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.766290 | controller |     - '        '
2026-03-14 09:56:26.766294 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.766298 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.766303 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.766307 | controller |     - '          exit 1'
2026-03-14 09:56:26.766311 | controller |     - '        fi'
2026-03-14 09:56:26.766316 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.766320 | controller |     - '        '
2026-03-14 09:56:26.766324 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.766334 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.766339 | controller |       | base64 -d)'
2026-03-14 09:56:26.766343 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.766348 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.766352 | controller |     - '          exit 1'
2026-03-14 09:56:26.766357 | controller |     - '        fi'
2026-03-14 09:56:26.766361 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.766366 | controller |     - '        '
2026-03-14 09:56:26.766370 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.766375 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.766379 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.766383 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.766388 | controller |       --timeout=300s'
2026-03-14 09:56:26.766392 | controller |     - '        '
2026-03-14 09:56:26.766396 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.766401 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.766407 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.766412 | controller |     - '        '
2026-03-14 09:56:26.766416 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.766420 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.766425 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.766429 | controller |     - '        '
2026-03-14 09:56:26.766433 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.766438 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.766442 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.766446 | controller |       available"'
2026-03-14 09:56:26.766450 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.766455 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.766459 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.766463 | controller |     - '          exit 1'
2026-03-14 09:56:26.766468 | controller |     - '        fi'
2026-03-14 09:56:26.766472 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.766476 | controller |     - '        '
2026-03-14 09:56:26.766481 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.766485 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.766489 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.766496 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.766500 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.766504 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.766511 | controller |       -- \'
2026-03-14 09:56:26.766515 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.766519 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.766523 | controller |     - '    logger.go:42: 09:45:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.766528 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.766532 | controller |     - '    logger.go:42: 09:45:57 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.766536 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.766541 | controller |       found'
2026-03-14 09:56:26.766545 | controller |     - '    logger.go:42: 09:45:58 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.766549 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.766554 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.766558 | controller |     - '        '
2026-03-14 09:56:26.766562 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.766567 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.766571 | controller |     - '        '
2026-03-14 09:56:26.766575 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.766581 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.766586 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.766590 | controller |     - '          exit 1'
2026-03-14 09:56:26.766606 | controller |     - '        fi'
2026-03-14 09:56:26.766613 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.766618 | controller |     - '        '
2026-03-14 09:56:26.766624 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.766631 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.766637 | controller |       | base64 -d)'
2026-03-14 09:56:26.766651 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.766656 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.766661 | controller |     - '          exit 1'
2026-03-14 09:56:26.766665 | controller |     - '        fi'
2026-03-14 09:56:26.766670 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.766674 | controller |     - '        '
2026-03-14 09:56:26.766678 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.766683 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.766687 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.766691 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.766696 | controller |       --timeout=300s'
2026-03-14 09:56:26.766700 | controller |     - '        '
2026-03-14 09:56:26.766704 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.766708 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.766713 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.766717 | controller |     - '        '
2026-03-14 09:56:26.766725 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.766729 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.766733 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.766738 | controller |     - '        '
2026-03-14 09:56:26.766742 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.766746 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.766751 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.766755 | controller |       available"'
2026-03-14 09:56:26.766762 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.766766 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.766771 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.766775 | controller |     - '          exit 1'
2026-03-14 09:56:26.766779 | controller |     - '        fi'
2026-03-14 09:56:26.766783 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.766788 | controller |     - '        '
2026-03-14 09:56:26.766792 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.766796 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.766801 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.766805 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.766809 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.766814 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.766818 | controller |       -- \'
2026-03-14 09:56:26.766822 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.766826 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.766831 | controller |     - '    logger.go:42: 09:45:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.766835 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.766839 | controller |     - '    logger.go:42: 09:45:58 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.766843 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.766848 | controller |       found'
2026-03-14 09:56:26.766852 | controller |     - '    logger.go:42: 09:45:59 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.766856 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.766861 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.766865 | controller |     - '        '
2026-03-14 09:56:26.766869 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.766873 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.766878 | controller |     - '        '
2026-03-14 09:56:26.766882 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.766886 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.766891 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.766895 | controller |     - '          exit 1'
2026-03-14 09:56:26.766899 | controller |     - '        fi'
2026-03-14 09:56:26.766903 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.766908 | controller |     - '        '
2026-03-14 09:56:26.766915 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.766920 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.766924 | controller |       | base64 -d)'
2026-03-14 09:56:26.766928 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.766941 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.766946 | controller |     - '          exit 1'
2026-03-14 09:56:26.766950 | controller |     - '        fi'
2026-03-14 09:56:26.766954 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.766959 | controller |     - '        '
2026-03-14 09:56:26.766963 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.766967 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.766972 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.766976 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.766980 | controller |       --timeout=300s'
2026-03-14 09:56:26.766984 | controller |     - '        '
2026-03-14 09:56:26.766989 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.766993 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.766997 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.767002 | controller |     - '        '
2026-03-14 09:56:26.767006 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.767010 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.767015 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.767031 | controller |     - '        '
2026-03-14 09:56:26.767037 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.767043 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.767047 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.767052 | controller |       available"'
2026-03-14 09:56:26.767056 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.767060 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.767064 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.767069 | controller |     - '          exit 1'
2026-03-14 09:56:26.767073 | controller |     - '        fi'
2026-03-14 09:56:26.767077 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.767082 | controller |     - '        '
2026-03-14 09:56:26.767086 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.767090 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.767095 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.767099 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.767103 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.767108 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.767112 | controller |       -- \'
2026-03-14 09:56:26.767116 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.767123 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.767127 | controller |     - '    logger.go:42: 09:45:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.767132 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.767136 | controller |     - '    logger.go:42: 09:45:59 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.767140 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.767145 | controller |       found'
2026-03-14 09:56:26.767149 | controller |     - '    logger.go:42: 09:46:00 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.767153 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.767158 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.767162 | controller |     - '        '
2026-03-14 09:56:26.767166 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.767170 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.767175 | controller |     - '        '
2026-03-14 09:56:26.767179 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.767183 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.767188 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.767192 | controller |     - '          exit 1'
2026-03-14 09:56:26.767196 | controller |     - '        fi'
2026-03-14 09:56:26.767201 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.767205 | controller |     - '        '
2026-03-14 09:56:26.767209 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.767217 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.767221 | controller |       | base64 -d)'
2026-03-14 09:56:26.767225 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.767238 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.767243 | controller |     - '          exit 1'
2026-03-14 09:56:26.767247 | controller |     - '        fi'
2026-03-14 09:56:26.767252 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.767256 | controller |     - '        '
2026-03-14 09:56:26.767260 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.767264 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.767269 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.767273 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.767277 | controller |       --timeout=300s'
2026-03-14 09:56:26.767281 | controller |     - '        '
2026-03-14 09:56:26.767286 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.767290 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.767294 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.767298 | controller |     - '        '
2026-03-14 09:56:26.767303 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.767307 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.767311 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.767315 | controller |     - '        '
2026-03-14 09:56:26.767320 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.767326 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.767330 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.767334 | controller |       available"'
2026-03-14 09:56:26.767338 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.767342 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.767347 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.767351 | controller |     - '          exit 1'
2026-03-14 09:56:26.767355 | controller |     - '        fi'
2026-03-14 09:56:26.767359 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.767364 | controller |     - '        '
2026-03-14 09:56:26.767368 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.767372 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.767376 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.767382 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.767387 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.767391 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.767395 | controller |       -- \'
2026-03-14 09:56:26.767399 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.767403 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.767408 | controller |     - '    logger.go:42: 09:46:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.767412 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.767416 | controller |     - '    logger.go:42: 09:46:00 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.767420 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.767424 | controller |       found'
2026-03-14 09:56:26.767429 | controller |     - '    logger.go:42: 09:46:01 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.767433 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.767437 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.767441 | controller |     - '        '
2026-03-14 09:56:26.767446 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.767450 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.767454 | controller |     - '        '
2026-03-14 09:56:26.767458 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.767462 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.767467 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.767471 | controller |     - '          exit 1'
2026-03-14 09:56:26.767475 | controller |     - '        fi'
2026-03-14 09:56:26.767479 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.767483 | controller |     - '        '
2026-03-14 09:56:26.767488 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.767492 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.767496 | controller |       | base64 -d)'
2026-03-14 09:56:26.767500 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.767505 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.767509 | controller |     - '          exit 1'
2026-03-14 09:56:26.767519 | controller |     - '        fi'
2026-03-14 09:56:26.767525 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.767529 | controller |     - '        '
2026-03-14 09:56:26.767534 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.767538 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.767542 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.767546 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.767551 | controller |       --timeout=300s'
2026-03-14 09:56:26.767555 | controller |     - '        '
2026-03-14 09:56:26.767559 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.767563 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.767568 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.767572 | controller |     - '        '
2026-03-14 09:56:26.767576 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.767580 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.767585 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.767589 | controller |     - '        '
2026-03-14 09:56:26.767604 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.767611 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.767615 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.767620 | controller |       available"'
2026-03-14 09:56:26.767624 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.767628 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.767633 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.767639 | controller |     - '          exit 1'
2026-03-14 09:56:26.767645 | controller |     - '        fi'
2026-03-14 09:56:26.767651 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.767658 | controller |     - '        '
2026-03-14 09:56:26.767663 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.767667 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.767671 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.767675 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.767679 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.767684 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.767688 | controller |       -- \'
2026-03-14 09:56:26.767692 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.767697 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.767701 | controller |     - '    logger.go:42: 09:46:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.767705 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.767709 | controller |     - '    logger.go:42: 09:46:01 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.767713 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.767728 | controller |       found'
2026-03-14 09:56:26.767732 | controller |     - '    logger.go:42: 09:46:02 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.767737 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.767741 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.767745 | controller |     - '        '
2026-03-14 09:56:26.767749 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.767754 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.767758 | controller |     - '        '
2026-03-14 09:56:26.767762 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.767766 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.767770 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.767775 | controller |     - '          exit 1'
2026-03-14 09:56:26.767779 | controller |     - '        fi'
2026-03-14 09:56:26.767783 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.767787 | controller |     - '        '
2026-03-14 09:56:26.767792 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.767796 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.767800 | controller |       | base64 -d)'
2026-03-14 09:56:26.767804 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.767808 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.767813 | controller |     - '          exit 1'
2026-03-14 09:56:26.767817 | controller |     - '        fi'
2026-03-14 09:56:26.767829 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.767836 | controller |     - '        '
2026-03-14 09:56:26.767841 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.767845 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.767849 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.767855 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.767860 | controller |       --timeout=300s'
2026-03-14 09:56:26.767864 | controller |     - '        '
2026-03-14 09:56:26.767868 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.767874 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.767878 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.767883 | controller |     - '        '
2026-03-14 09:56:26.767887 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.767891 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.767895 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.767900 | controller |     - '        '
2026-03-14 09:56:26.767904 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.767908 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.767912 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.767917 | controller |       available"'
2026-03-14 09:56:26.767921 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.767925 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.767929 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.767934 | controller |     - '          exit 1'
2026-03-14 09:56:26.767939 | controller |     - '        fi'
2026-03-14 09:56:26.767944 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.767948 | controller |     - '        '
2026-03-14 09:56:26.767952 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.767956 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.767961 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.767965 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.767969 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.767973 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.767978 | controller |       -- \'
2026-03-14 09:56:26.767982 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.767986 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.767990 | controller |     - '    logger.go:42: 09:46:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.767995 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.767999 | controller |     - '    logger.go:42: 09:46:03 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.768003 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.768007 | controller |       found'
2026-03-14 09:56:26.768012 | controller |     - '    logger.go:42: 09:46:04 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.768016 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.768020 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.768024 | controller |     - '        '
2026-03-14 09:56:26.768029 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.768033 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.768037 | controller |     - '        '
2026-03-14 09:56:26.768041 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.768045 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.768052 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.768056 | controller |     - '          exit 1'
2026-03-14 09:56:26.768060 | controller |     - '        fi'
2026-03-14 09:56:26.768065 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.768069 | controller |     - '        '
2026-03-14 09:56:26.768073 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.768078 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.768082 | controller |       | base64 -d)'
2026-03-14 09:56:26.768086 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.768090 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.768095 | controller |     - '          exit 1'
2026-03-14 09:56:26.768099 | controller |     - '        fi'
2026-03-14 09:56:26.768103 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.768108 | controller |     - '        '
2026-03-14 09:56:26.768118 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.768122 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.768127 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.768131 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.768135 | controller |       --timeout=300s'
2026-03-14 09:56:26.768158 | controller |     - '        '
2026-03-14 09:56:26.768163 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.768167 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.768171 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.768176 | controller |     - '        '
2026-03-14 09:56:26.768180 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.768184 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.768189 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.768193 | controller |     - '        '
2026-03-14 09:56:26.768197 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.768201 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.768206 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.768210 | controller |       available"'
2026-03-14 09:56:26.768214 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.768218 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.768223 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.768227 | controller |     - '          exit 1'
2026-03-14 09:56:26.768231 | controller |     - '        fi'
2026-03-14 09:56:26.768236 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.768240 | controller |     - '        '
2026-03-14 09:56:26.768244 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.768248 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.768253 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.768257 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.768261 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.768266 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.768270 | controller |       -- \'
2026-03-14 09:56:26.768274 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.768279 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.768283 | controller |     - '    logger.go:42: 09:46:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.768287 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.768292 | controller |     - '    logger.go:42: 09:46:04 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.768296 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.768300 | controller |       found'
2026-03-14 09:56:26.768305 | controller |     - '    logger.go:42: 09:46:05 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.768309 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.768313 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.768318 | controller |     - '        '
2026-03-14 09:56:26.768322 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.768326 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.768330 | controller |     - '        '
2026-03-14 09:56:26.768338 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.768342 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.768346 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.768351 | controller |     - '          exit 1'
2026-03-14 09:56:26.768355 | controller |     - '        fi'
2026-03-14 09:56:26.768359 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.768363 | controller |     - '        '
2026-03-14 09:56:26.768368 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.768372 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.768376 | controller |       | base64 -d)'
2026-03-14 09:56:26.768380 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.768385 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.768389 | controller |     - '          exit 1'
2026-03-14 09:56:26.768393 | controller |     - '        fi'
2026-03-14 09:56:26.768398 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.768402 | controller |     - '        '
2026-03-14 09:56:26.768406 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.768417 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.768422 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.768426 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.768437 | controller |       --timeout=300s'
2026-03-14 09:56:26.768441 | controller |     - '        '
2026-03-14 09:56:26.768445 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.768450 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.768454 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.768460 | controller |     - '        '
2026-03-14 09:56:26.768464 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.768468 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.768473 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.768477 | controller |     - '        '
2026-03-14 09:56:26.768481 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.768485 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.768490 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.768494 | controller |       available"'
2026-03-14 09:56:26.768498 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.768502 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.768508 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.768512 | controller |     - '          exit 1'
2026-03-14 09:56:26.768517 | controller |     - '        fi'
2026-03-14 09:56:26.768521 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.768525 | controller |     - '        '
2026-03-14 09:56:26.768529 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.768533 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.768538 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.768543 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.768548 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.768552 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.768556 | controller |       -- \'
2026-03-14 09:56:26.768561 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.768565 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.768569 | controller |     - '    logger.go:42: 09:46:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.768573 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.768578 | controller |     - '    logger.go:42: 09:46:05 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.768582 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.768586 | controller |       found'
2026-03-14 09:56:26.768590 | controller |     - '    logger.go:42: 09:46:06 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.768607 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.768613 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.768617 | controller |     - '        '
2026-03-14 09:56:26.768622 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.768626 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.768630 | controller |     - '        '
2026-03-14 09:56:26.768634 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.768639 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.768643 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.768648 | controller |     - '          exit 1'
2026-03-14 09:56:26.768654 | controller |     - '        fi'
2026-03-14 09:56:26.768660 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.768667 | controller |     - '        '
2026-03-14 09:56:26.768673 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.768678 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.768683 | controller |       | base64 -d)'
2026-03-14 09:56:26.768687 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.768691 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.768696 | controller |     - '          exit 1'
2026-03-14 09:56:26.768700 | controller |     - '        fi'
2026-03-14 09:56:26.768704 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.768708 | controller |     - '        '
2026-03-14 09:56:26.768713 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.768725 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.768729 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.768734 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.768738 | controller |       --timeout=300s'
2026-03-14 09:56:26.768742 | controller |     - '        '
2026-03-14 09:56:26.768747 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.768751 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.768755 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.768760 | controller |     - '        '
2026-03-14 09:56:26.768764 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.768772 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.768793 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.768798 | controller |     - '        '
2026-03-14 09:56:26.768802 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.768806 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.768811 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.768815 | controller |       available"'
2026-03-14 09:56:26.768819 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.768824 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.768828 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.768832 | controller |     - '          exit 1'
2026-03-14 09:56:26.768836 | controller |     - '        fi'
2026-03-14 09:56:26.768841 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.768845 | controller |     - '        '
2026-03-14 09:56:26.768849 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.768853 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.768858 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.768862 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.768866 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.768870 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.768875 | controller |       -- \'
2026-03-14 09:56:26.768879 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.768883 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.768888 | controller |     - '    logger.go:42: 09:46:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.768892 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.768896 | controller |     - '    logger.go:42: 09:46:06 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.768900 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.768905 | controller |       found'
2026-03-14 09:56:26.768909 | controller |     - '    logger.go:42: 09:46:07 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.768913 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.768918 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.768922 | controller |     - '        '
2026-03-14 09:56:26.768926 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.768930 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.768935 | controller |     - '        '
2026-03-14 09:56:26.768939 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.768943 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.768948 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.768952 | controller |     - '          exit 1'
2026-03-14 09:56:26.768956 | controller |     - '        fi'
2026-03-14 09:56:26.768963 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.768967 | controller |     - '        '
2026-03-14 09:56:26.768972 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.768977 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.768982 | controller |       | base64 -d)'
2026-03-14 09:56:26.768986 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.768990 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.768995 | controller |     - '          exit 1'
2026-03-14 09:56:26.768999 | controller |     - '        fi'
2026-03-14 09:56:26.769003 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.769008 | controller |     - '        '
2026-03-14 09:56:26.769012 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.769016 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.769027 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.769032 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.769037 | controller |       --timeout=300s'
2026-03-14 09:56:26.769041 | controller |     - '        '
2026-03-14 09:56:26.769046 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.769050 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.769054 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.769059 | controller |     - '        '
2026-03-14 09:56:26.769063 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.769067 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.769072 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.769076 | controller |     - '        '
2026-03-14 09:56:26.769081 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.769087 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.769091 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.769096 | controller |       available"'
2026-03-14 09:56:26.769100 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.769104 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.769109 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.769113 | controller |     - '          exit 1'
2026-03-14 09:56:26.769117 | controller |     - '        fi'
2026-03-14 09:56:26.769122 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.769126 | controller |     - '        '
2026-03-14 09:56:26.769130 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.769135 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.769139 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.769143 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.769148 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.769152 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.769156 | controller |       -- \'
2026-03-14 09:56:26.769161 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.769167 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.769171 | controller |     - '    logger.go:42: 09:46:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.769176 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.769180 | controller |     - '    logger.go:42: 09:46:07 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.769185 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.769189 | controller |       found'
2026-03-14 09:56:26.769193 | controller |     - '    logger.go:42: 09:46:08 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.769198 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.769202 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.769206 | controller |     - '        '
2026-03-14 09:56:26.769211 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.769215 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.769219 | controller |     - '        '
2026-03-14 09:56:26.769224 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.769228 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.769232 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.769237 | controller |     - '          exit 1'
2026-03-14 09:56:26.769241 | controller |     - '        fi'
2026-03-14 09:56:26.769245 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.769251 | controller |     - '        '
2026-03-14 09:56:26.769256 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.769260 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.769266 | controller |       | base64 -d)'
2026-03-14 09:56:26.769270 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.769275 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.769279 | controller |     - '          exit 1'
2026-03-14 09:56:26.769283 | controller |     - '        fi'
2026-03-14 09:56:26.769288 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.769292 | controller |     - '        '
2026-03-14 09:56:26.769296 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.769300 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.769312 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.769316 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.769321 | controller |       --timeout=300s'
2026-03-14 09:56:26.769325 | controller |     - '        '
2026-03-14 09:56:26.769329 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.769334 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.769338 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.769343 | controller |     - '        '
2026-03-14 09:56:26.769347 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.769353 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.769357 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.769362 | controller |     - '        '
2026-03-14 09:56:26.769366 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.769371 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.769377 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.769381 | controller |       available"'
2026-03-14 09:56:26.769385 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.769390 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.769394 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.769398 | controller |     - '          exit 1'
2026-03-14 09:56:26.769403 | controller |     - '        fi'
2026-03-14 09:56:26.769407 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.769411 | controller |     - '        '
2026-03-14 09:56:26.769416 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.769420 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.769424 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.769429 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.769433 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.769439 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.769444 | controller |       -- \'
2026-03-14 09:56:26.769448 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.769453 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.769457 | controller |     - '    logger.go:42: 09:46:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.769461 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.769466 | controller |     - '    logger.go:42: 09:46:08 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.769470 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.769474 | controller |       found'
2026-03-14 09:56:26.769479 | controller |     - '    logger.go:42: 09:46:09 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.769483 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.769487 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.769492 | controller |     - '        '
2026-03-14 09:56:26.769496 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.769500 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.769505 | controller |     - '        '
2026-03-14 09:56:26.769509 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.769513 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.769518 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.769522 | controller |     - '          exit 1'
2026-03-14 09:56:26.769526 | controller |     - '        fi'
2026-03-14 09:56:26.769531 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.769535 | controller |     - '        '
2026-03-14 09:56:26.769539 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.769543 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.769548 | controller |       | base64 -d)'
2026-03-14 09:56:26.769552 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.769556 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.769561 | controller |     - '          exit 1'
2026-03-14 09:56:26.769565 | controller |     - '        fi'
2026-03-14 09:56:26.769569 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.769577 | controller |     - '        '
2026-03-14 09:56:26.769581 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.769586 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.769590 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.769613 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.769620 | controller |       --timeout=300s'
2026-03-14 09:56:26.769624 | controller |     - '        '
2026-03-14 09:56:26.769629 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.769633 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.769638 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.769642 | controller |     - '        '
2026-03-14 09:56:26.769646 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.769651 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.769655 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.769659 | controller |     - '        '
2026-03-14 09:56:26.769664 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.769671 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.769677 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.769687 | controller |       available"'
2026-03-14 09:56:26.769693 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.769698 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.769702 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.769706 | controller |     - '          exit 1'
2026-03-14 09:56:26.769711 | controller |     - '        fi'
2026-03-14 09:56:26.769715 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.769719 | controller |     - '        '
2026-03-14 09:56:26.769724 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.769728 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.769732 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.769737 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.769741 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.769745 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.769750 | controller |       -- \'
2026-03-14 09:56:26.769755 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.769761 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.769768 | controller |     - '    logger.go:42: 09:46:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.769773 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.769778 | controller |     - '    logger.go:42: 09:46:09 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.769782 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.769789 | controller |       found'
2026-03-14 09:56:26.769795 | controller |     - '    logger.go:42: 09:46:10 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.769803 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.769808 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.769812 | controller |     - '        '
2026-03-14 09:56:26.769816 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.769821 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.769825 | controller |     - '        '
2026-03-14 09:56:26.769829 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.769834 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.769838 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.769842 | controller |     - '          exit 1'
2026-03-14 09:56:26.769847 | controller |     - '        fi'
2026-03-14 09:56:26.769851 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.769855 | controller |     - '        '
2026-03-14 09:56:26.769860 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.769866 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.769871 | controller |       | base64 -d)'
2026-03-14 09:56:26.769875 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.769879 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.769884 | controller |     - '          exit 1'
2026-03-14 09:56:26.769888 | controller |     - '        fi'
2026-03-14 09:56:26.769892 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.769897 | controller |     - '        '
2026-03-14 09:56:26.769901 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.769906 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.769910 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.769923 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.769928 | controller |       --timeout=300s'
2026-03-14 09:56:26.769932 | controller |     - '        '
2026-03-14 09:56:26.769937 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.769942 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.769948 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.769954 | controller |     - '        '
2026-03-14 09:56:26.769961 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.769966 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.769972 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.769978 | controller |     - '        '
2026-03-14 09:56:26.769984 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.769989 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.769993 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.769998 | controller |       available"'
2026-03-14 09:56:26.770002 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.770007 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.770011 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.770017 | controller |     - '          exit 1'
2026-03-14 09:56:26.770024 | controller |     - '        fi'
2026-03-14 09:56:26.770036 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.770042 | controller |     - '        '
2026-03-14 09:56:26.770048 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.770054 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.770058 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.770063 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.770067 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.770072 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.770076 | controller |       -- \'
2026-03-14 09:56:26.770081 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.770085 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.770089 | controller |     - '    logger.go:42: 09:46:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.770094 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.770098 | controller |     - '    logger.go:42: 09:46:11 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.770102 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.770107 | controller |       found'
2026-03-14 09:56:26.770111 | controller |     - '    logger.go:42: 09:46:12 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.770115 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.770120 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.770124 | controller |     - '        '
2026-03-14 09:56:26.770128 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.770132 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.770137 | controller |     - '        '
2026-03-14 09:56:26.770141 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.770145 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.770150 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.770154 | controller |     - '          exit 1'
2026-03-14 09:56:26.770158 | controller |     - '        fi'
2026-03-14 09:56:26.770163 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.770167 | controller |     - '        '
2026-03-14 09:56:26.770171 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.770175 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.770180 | controller |       | base64 -d)'
2026-03-14 09:56:26.770184 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.770188 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.770193 | controller |     - '          exit 1'
2026-03-14 09:56:26.770197 | controller |     - '        fi'
2026-03-14 09:56:26.770201 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.770205 | controller |     - '        '
2026-03-14 09:56:26.770210 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.770214 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.770218 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.770223 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.770227 | controller |       --timeout=300s'
2026-03-14 09:56:26.770231 | controller |     - '        '
2026-03-14 09:56:26.770246 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.770251 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.770256 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.770260 | controller |     - '        '
2026-03-14 09:56:26.770264 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.770268 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.770273 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.770277 | controller |     - '        '
2026-03-14 09:56:26.770281 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.770286 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.770290 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.770294 | controller |       available"'
2026-03-14 09:56:26.770298 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.770303 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.770307 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.770311 | controller |     - '          exit 1'
2026-03-14 09:56:26.770316 | controller |     - '        fi'
2026-03-14 09:56:26.770320 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.770326 | controller |     - '        '
2026-03-14 09:56:26.770331 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.770335 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.770339 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.770343 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.770348 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.770352 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.770356 | controller |       -- \'
2026-03-14 09:56:26.770360 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.770365 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.770369 | controller |     - '    logger.go:42: 09:46:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.770373 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.770378 | controller |     - '    logger.go:42: 09:46:12 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.770382 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.770386 | controller |       found'
2026-03-14 09:56:26.770391 | controller |     - '    logger.go:42: 09:46:13 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.770395 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.770399 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.770403 | controller |     - '        '
2026-03-14 09:56:26.770408 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.770412 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.770416 | controller |     - '        '
2026-03-14 09:56:26.770420 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.770426 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.770431 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.770435 | controller |     - '          exit 1'
2026-03-14 09:56:26.770439 | controller |     - '        fi'
2026-03-14 09:56:26.770443 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.770448 | controller |     - '        '
2026-03-14 09:56:26.770452 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.770456 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.770460 | controller |       | base64 -d)'
2026-03-14 09:56:26.770465 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.770469 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.770473 | controller |     - '          exit 1'
2026-03-14 09:56:26.770478 | controller |     - '        fi'
2026-03-14 09:56:26.770482 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.770486 | controller |     - '        '
2026-03-14 09:56:26.770490 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.770495 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.770499 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.770503 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.770507 | controller |       --timeout=300s'
2026-03-14 09:56:26.770512 | controller |     - '        '
2026-03-14 09:56:26.770522 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.770526 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.770531 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.770535 | controller |     - '        '
2026-03-14 09:56:26.770539 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.770543 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.770550 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.770554 | controller |     - '        '
2026-03-14 09:56:26.770558 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.770563 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.770567 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.770571 | controller |       available"'
2026-03-14 09:56:26.770576 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.770580 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.770584 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.770588 | controller |     - '          exit 1'
2026-03-14 09:56:26.770604 | controller |     - '        fi'
2026-03-14 09:56:26.770612 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.770616 | controller |     - '        '
2026-03-14 09:56:26.770621 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.770625 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.770629 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.770634 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.770640 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.770644 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.770650 | controller |       -- \'
2026-03-14 09:56:26.770655 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.770659 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.770663 | controller |     - '    logger.go:42: 09:46:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.770668 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.770672 | controller |     - '    logger.go:42: 09:46:13 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.770676 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.770681 | controller |       found'
2026-03-14 09:56:26.770687 | controller |     - '    logger.go:42: 09:46:14 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.770694 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.770700 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.770706 | controller |     - '        '
2026-03-14 09:56:26.770712 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.770716 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.770721 | controller |     - '        '
2026-03-14 09:56:26.770725 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.770729 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.770734 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.770738 | controller |     - '          exit 1'
2026-03-14 09:56:26.770742 | controller |     - '        fi'
2026-03-14 09:56:26.770746 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.770751 | controller |     - '        '
2026-03-14 09:56:26.770755 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.770759 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.770763 | controller |       | base64 -d)'
2026-03-14 09:56:26.770768 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.770772 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.770776 | controller |     - '          exit 1'
2026-03-14 09:56:26.770781 | controller |     - '        fi'
2026-03-14 09:56:26.770785 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.770789 | controller |     - '        '
2026-03-14 09:56:26.770793 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.770798 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.770802 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.770806 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.770810 | controller |       --timeout=300s'
2026-03-14 09:56:26.770815 | controller |     - '        '
2026-03-14 09:56:26.770819 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.770831 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.770836 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.770840 | controller |     - '        '
2026-03-14 09:56:26.770844 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.770851 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.770855 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.770860 | controller |     - '        '
2026-03-14 09:56:26.770864 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.770868 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.770872 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.770877 | controller |       available"'
2026-03-14 09:56:26.770881 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.770892 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.770896 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.770901 | controller |     - '          exit 1'
2026-03-14 09:56:26.770905 | controller |     - '        fi'
2026-03-14 09:56:26.770913 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.770918 | controller |     - '        '
2026-03-14 09:56:26.770922 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.770929 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.770933 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.770937 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.770941 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.770946 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.770950 | controller |       -- \'
2026-03-14 09:56:26.770954 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.770959 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.770963 | controller |     - '    logger.go:42: 09:46:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.770967 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.770971 | controller |     - '    logger.go:42: 09:46:14 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.770976 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.770980 | controller |       found'
2026-03-14 09:56:26.770986 | controller |     - '    logger.go:42: 09:46:15 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.770990 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.770995 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.770999 | controller |     - '        '
2026-03-14 09:56:26.771003 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.771007 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.771011 | controller |     - '        '
2026-03-14 09:56:26.771016 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.771037 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.771044 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.771048 | controller |     - '          exit 1'
2026-03-14 09:56:26.771053 | controller |     - '        fi'
2026-03-14 09:56:26.771057 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.771061 | controller |     - '        '
2026-03-14 09:56:26.771065 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.771070 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.771076 | controller |       | base64 -d)'
2026-03-14 09:56:26.771081 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.771085 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.771089 | controller |     - '          exit 1'
2026-03-14 09:56:26.771094 | controller |     - '        fi'
2026-03-14 09:56:26.771098 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.771102 | controller |     - '        '
2026-03-14 09:56:26.771106 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.771111 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.771115 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.771119 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.771123 | controller |       --timeout=300s'
2026-03-14 09:56:26.771128 | controller |     - '        '
2026-03-14 09:56:26.771135 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.771147 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.771151 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.771156 | controller |     - '        '
2026-03-14 09:56:26.771160 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.771164 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.771169 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.771173 | controller |     - '        '
2026-03-14 09:56:26.771177 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.771181 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.771186 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.771190 | controller |       available"'
2026-03-14 09:56:26.771194 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.771198 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.771205 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.771209 | controller |     - '          exit 1'
2026-03-14 09:56:26.771213 | controller |     - '        fi'
2026-03-14 09:56:26.771218 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.771222 | controller |     - '        '
2026-03-14 09:56:26.771226 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.771230 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.771235 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.771239 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.771243 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.771247 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.771252 | controller |       -- \'
2026-03-14 09:56:26.771257 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.771261 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.771267 | controller |     - '    logger.go:42: 09:46:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.771271 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.771276 | controller |     - '    logger.go:42: 09:46:15 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.771280 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.771284 | controller |       found'
2026-03-14 09:56:26.771293 | controller |     - '    logger.go:42: 09:46:16 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.771298 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.771302 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.771307 | controller |     - '        '
2026-03-14 09:56:26.771311 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.771315 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.771319 | controller |     - '        '
2026-03-14 09:56:26.771324 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.771328 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.771332 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.771337 | controller |     - '          exit 1'
2026-03-14 09:56:26.771342 | controller |     - '        fi'
2026-03-14 09:56:26.771348 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.771352 | controller |     - '        '
2026-03-14 09:56:26.771357 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.771362 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.771367 | controller |       | base64 -d)'
2026-03-14 09:56:26.771371 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.771376 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.771380 | controller |     - '          exit 1'
2026-03-14 09:56:26.771384 | controller |     - '        fi'
2026-03-14 09:56:26.771389 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.771393 | controller |     - '        '
2026-03-14 09:56:26.771397 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.771401 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.771406 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.771410 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.771414 | controller |       --timeout=300s'
2026-03-14 09:56:26.771418 | controller |     - '        '
2026-03-14 09:56:26.771423 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.771433 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.771437 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.771442 | controller |     - '        '
2026-03-14 09:56:26.771446 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.771450 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.771454 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.771459 | controller |     - '        '
2026-03-14 09:56:26.771463 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.771467 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.771472 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.771478 | controller |       available"'
2026-03-14 09:56:26.771488 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.771492 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.771497 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.771501 | controller |     - '          exit 1'
2026-03-14 09:56:26.771505 | controller |     - '        fi'
2026-03-14 09:56:26.771514 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.771518 | controller |     - '        '
2026-03-14 09:56:26.771522 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.771527 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.771531 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.771535 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.771542 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.771547 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.771553 | controller |       -- \'
2026-03-14 09:56:26.771559 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.771567 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.771573 | controller |     - '    logger.go:42: 09:46:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.771577 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.771583 | controller |     - '    logger.go:42: 09:46:16 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.771589 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.771609 | controller |       found'
2026-03-14 09:56:26.771616 | controller |     - '    logger.go:42: 09:46:17 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.771621 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.771625 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.771629 | controller |     - '        '
2026-03-14 09:56:26.771634 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.771638 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.771642 | controller |     - '        '
2026-03-14 09:56:26.771646 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.771651 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.771655 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.771665 | controller |     - '          exit 1'
2026-03-14 09:56:26.771669 | controller |     - '        fi'
2026-03-14 09:56:26.771674 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.771680 | controller |     - '        '
2026-03-14 09:56:26.771685 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.771692 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.771696 | controller |       | base64 -d)'
2026-03-14 09:56:26.771703 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.771708 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.771713 | controller |     - '          exit 1'
2026-03-14 09:56:26.771719 | controller |     - '        fi'
2026-03-14 09:56:26.771725 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.771730 | controller |     - '        '
2026-03-14 09:56:26.771737 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.771742 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.771748 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.771754 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.771759 | controller |       --timeout=300s'
2026-03-14 09:56:26.771764 | controller |     - '        '
2026-03-14 09:56:26.771768 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.771772 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.771786 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.771790 | controller |     - '        '
2026-03-14 09:56:26.771795 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.771799 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.771803 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.771808 | controller |     - '        '
2026-03-14 09:56:26.771812 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.771816 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.771821 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.771825 | controller |       available"'
2026-03-14 09:56:26.771829 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.771833 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.771838 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.771842 | controller |     - '          exit 1'
2026-03-14 09:56:26.771846 | controller |     - '        fi'
2026-03-14 09:56:26.771850 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.771855 | controller |     - '        '
2026-03-14 09:56:26.771859 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.771863 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.771870 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.771876 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.771882 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.771888 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.771893 | controller |       -- \'
2026-03-14 09:56:26.771899 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.771905 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.771910 | controller |     - '    logger.go:42: 09:46:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.771915 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.771921 | controller |     - '    logger.go:42: 09:46:18 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.771927 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.771932 | controller |       found'
2026-03-14 09:56:26.771937 | controller |     - '    logger.go:42: 09:46:19 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.771945 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.771953 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.771958 | controller |     - '        '
2026-03-14 09:56:26.771962 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.771967 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.771971 | controller |     - '        '
2026-03-14 09:56:26.771976 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.771980 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.771984 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.771989 | controller |     - '          exit 1'
2026-03-14 09:56:26.771993 | controller |     - '        fi'
2026-03-14 09:56:26.771997 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.772002 | controller |     - '        '
2026-03-14 09:56:26.772006 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.772010 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.772015 | controller |       | base64 -d)'
2026-03-14 09:56:26.772019 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.772023 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.772027 | controller |     - '          exit 1'
2026-03-14 09:56:26.772032 | controller |     - '        fi'
2026-03-14 09:56:26.772036 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.772040 | controller |     - '        '
2026-03-14 09:56:26.772045 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.772049 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.772053 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.772058 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.772062 | controller |       --timeout=300s'
2026-03-14 09:56:26.772066 | controller |     - '        '
2026-03-14 09:56:26.772071 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.772078 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.772093 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.772098 | controller |     - '        '
2026-03-14 09:56:26.772104 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.772110 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.772115 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.772120 | controller |     - '        '
2026-03-14 09:56:26.772125 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.772131 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.772137 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.772141 | controller |       available"'
2026-03-14 09:56:26.772146 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.772150 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.772156 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.772162 | controller |     - '          exit 1'
2026-03-14 09:56:26.772167 | controller |     - '        fi'
2026-03-14 09:56:26.772172 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.772179 | controller |     - '        '
2026-03-14 09:56:26.772183 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.772188 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.772192 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.772196 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.772201 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.772205 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.772209 | controller |       -- \'
2026-03-14 09:56:26.772214 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.772218 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.772224 | controller |     - '    logger.go:42: 09:46:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.772229 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.772233 | controller |     - '    logger.go:42: 09:46:19 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.772237 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.772242 | controller |       found'
2026-03-14 09:56:26.772246 | controller |     - '    logger.go:42: 09:46:20 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.772250 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.772255 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.772259 | controller |     - '        '
2026-03-14 09:56:26.772263 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.772267 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.772272 | controller |     - '        '
2026-03-14 09:56:26.772276 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.772280 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.772285 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.772289 | controller |     - '          exit 1'
2026-03-14 09:56:26.772293 | controller |     - '        fi'
2026-03-14 09:56:26.772298 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.772302 | controller |     - '        '
2026-03-14 09:56:26.772308 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.772314 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.772320 | controller |       | base64 -d)'
2026-03-14 09:56:26.772337 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.772341 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.772347 | controller |     - '          exit 1'
2026-03-14 09:56:26.772353 | controller |     - '        fi'
2026-03-14 09:56:26.772358 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.772362 | controller |     - '        '
2026-03-14 09:56:26.772367 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.772373 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.772379 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.772384 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.772389 | controller |       --timeout=300s'
2026-03-14 09:56:26.772393 | controller |     - '        '
2026-03-14 09:56:26.772397 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.772404 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.772409 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.772413 | controller |     - '        '
2026-03-14 09:56:26.772425 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.772430 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.772434 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.772439 | controller |     - '        '
2026-03-14 09:56:26.772443 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.772447 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.772452 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.772456 | controller |       available"'
2026-03-14 09:56:26.772460 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.772465 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.772469 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.772473 | controller |     - '          exit 1'
2026-03-14 09:56:26.772477 | controller |     - '        fi'
2026-03-14 09:56:26.772482 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.772486 | controller |     - '        '
2026-03-14 09:56:26.772490 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.772494 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.772499 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.772503 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.772507 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.772512 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.772516 | controller |       -- \'
2026-03-14 09:56:26.772521 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.772527 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.772533 | controller |     - '    logger.go:42: 09:46:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.772539 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.772545 | controller |     - '    logger.go:42: 09:46:20 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.772551 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.772556 | controller |       found'
2026-03-14 09:56:26.772562 | controller |     - '    logger.go:42: 09:46:21 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.772568 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.772573 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.772578 | controller |     - '        '
2026-03-14 09:56:26.772584 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.772590 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.772607 | controller |     - '        '
2026-03-14 09:56:26.772614 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.772623 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.772627 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.772632 | controller |     - '          exit 1'
2026-03-14 09:56:26.772636 | controller |     - '        fi'
2026-03-14 09:56:26.772640 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.772645 | controller |     - '        '
2026-03-14 09:56:26.772649 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.772656 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.772660 | controller |       | base64 -d)'
2026-03-14 09:56:26.772665 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.772669 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.772673 | controller |     - '          exit 1'
2026-03-14 09:56:26.772678 | controller |     - '        fi'
2026-03-14 09:56:26.772682 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.772686 | controller |     - '        '
2026-03-14 09:56:26.772691 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.772695 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.772699 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.772703 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.772708 | controller |       --timeout=300s'
2026-03-14 09:56:26.772712 | controller |     - '        '
2026-03-14 09:56:26.772716 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.772722 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.772728 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.772733 | controller |     - '        '
2026-03-14 09:56:26.772747 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.772754 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.772760 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.772769 | controller |     - '        '
2026-03-14 09:56:26.772774 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.772779 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.772784 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.772790 | controller |       available"'
2026-03-14 09:56:26.772796 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.772800 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.772805 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.772809 | controller |     - '          exit 1'
2026-03-14 09:56:26.772813 | controller |     - '        fi'
2026-03-14 09:56:26.772818 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.772822 | controller |     - '        '
2026-03-14 09:56:26.772826 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.772830 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.772835 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.772839 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.772843 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.772859 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.772863 | controller |       -- \'
2026-03-14 09:56:26.772868 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.772872 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.772876 | controller |     - '    logger.go:42: 09:46:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.772881 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.772885 | controller |     - '    logger.go:42: 09:46:21 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.772891 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.772896 | controller |       found'
2026-03-14 09:56:26.772900 | controller |     - '    logger.go:42: 09:46:22 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.772904 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.772909 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.772913 | controller |     - '        '
2026-03-14 09:56:26.772917 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.772922 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.772926 | controller |     - '        '
2026-03-14 09:56:26.772932 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.772937 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.772943 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.772949 | controller |     - '          exit 1'
2026-03-14 09:56:26.772955 | controller |     - '        fi'
2026-03-14 09:56:26.772961 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.772966 | controller |     - '        '
2026-03-14 09:56:26.772972 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.772978 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.772984 | controller |       | base64 -d)'
2026-03-14 09:56:26.772988 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.772994 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.773000 | controller |     - '          exit 1'
2026-03-14 09:56:26.773006 | controller |     - '        fi'
2026-03-14 09:56:26.773010 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.773014 | controller |     - '        '
2026-03-14 09:56:26.773019 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.773023 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.773027 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.773031 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.773036 | controller |       --timeout=300s'
2026-03-14 09:56:26.773040 | controller |     - '        '
2026-03-14 09:56:26.773044 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.773049 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.773053 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.773057 | controller |     - '        '
2026-03-14 09:56:26.773062 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.773074 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.773081 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.773086 | controller |     - '        '
2026-03-14 09:56:26.773090 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.773094 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.773099 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.773103 | controller |       available"'
2026-03-14 09:56:26.773107 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.773112 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.773116 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.773120 | controller |     - '          exit 1'
2026-03-14 09:56:26.773124 | controller |     - '        fi'
2026-03-14 09:56:26.773129 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.773133 | controller |     - '        '
2026-03-14 09:56:26.773139 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.773145 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.773151 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.773156 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.773162 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.773168 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.773173 | controller |       -- \'
2026-03-14 09:56:26.773179 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.773185 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.773190 | controller |     - '    logger.go:42: 09:46:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.773194 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.773198 | controller |     - '    logger.go:42: 09:46:22 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.773203 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.773209 | controller |       found'
2026-03-14 09:56:26.773215 | controller |     - '    logger.go:42: 09:46:23 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.773220 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.773225 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.773229 | controller |     - '        '
2026-03-14 09:56:26.773235 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.773241 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.773247 | controller |     - '        '
2026-03-14 09:56:26.773252 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.773256 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.773262 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.773268 | controller |     - '          exit 1'
2026-03-14 09:56:26.773273 | controller |     - '        fi'
2026-03-14 09:56:26.773277 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.773282 | controller |     - '        '
2026-03-14 09:56:26.773286 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.773291 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.773300 | controller |       | base64 -d)'
2026-03-14 09:56:26.773306 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.773310 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.773314 | controller |     - '          exit 1'
2026-03-14 09:56:26.773319 | controller |     - '        fi'
2026-03-14 09:56:26.773323 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.773335 | controller |     - '        '
2026-03-14 09:56:26.773340 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.773344 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.773348 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.773354 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.773359 | controller |       --timeout=300s'
2026-03-14 09:56:26.773364 | controller |     - '        '
2026-03-14 09:56:26.773368 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.773374 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.773379 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.773383 | controller |     - '        '
2026-03-14 09:56:26.773388 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.773400 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.773404 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.773409 | controller |     - '        '
2026-03-14 09:56:26.773413 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.773418 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.773422 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.773426 | controller |       available"'
2026-03-14 09:56:26.773430 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.773435 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.773439 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.773444 | controller |     - '          exit 1'
2026-03-14 09:56:26.773450 | controller |     - '        fi'
2026-03-14 09:56:26.773456 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.773461 | controller |     - '        '
2026-03-14 09:56:26.773467 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.773473 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.773478 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.773488 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.773494 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.773498 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.773504 | controller |       -- \'
2026-03-14 09:56:26.773509 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.773515 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.773520 | controller |     - '    logger.go:42: 09:46:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.773528 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.773534 | controller |     - '    logger.go:42: 09:46:23 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.773540 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.773545 | controller |       found'
2026-03-14 09:56:26.773551 | controller |     - '    logger.go:42: 09:46:24 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.773557 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.773565 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.773571 | controller |     - '        '
2026-03-14 09:56:26.773577 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.773581 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.773586 | controller |     - '        '
2026-03-14 09:56:26.773591 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.773611 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.773616 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.773621 | controller |     - '          exit 1'
2026-03-14 09:56:26.773625 | controller |     - '        fi'
2026-03-14 09:56:26.773629 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.773636 | controller |     - '        '
2026-03-14 09:56:26.773640 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.773645 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.773658 | controller |       | base64 -d)'
2026-03-14 09:56:26.773663 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.773667 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.773671 | controller |     - '          exit 1'
2026-03-14 09:56:26.773676 | controller |     - '        fi'
2026-03-14 09:56:26.773680 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.773684 | controller |     - '        '
2026-03-14 09:56:26.773689 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.773693 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.773697 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.773701 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.773706 | controller |       --timeout=300s'
2026-03-14 09:56:26.773710 | controller |     - '        '
2026-03-14 09:56:26.773714 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.773720 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.773727 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.773731 | controller |     - '        '
2026-03-14 09:56:26.773736 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.773742 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.773757 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.773764 | controller |     - '        '
2026-03-14 09:56:26.773770 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.773776 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.773781 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.773790 | controller |       available"'
2026-03-14 09:56:26.773794 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.773798 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.773803 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.773807 | controller |     - '          exit 1'
2026-03-14 09:56:26.773811 | controller |     - '        fi'
2026-03-14 09:56:26.773815 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.773820 | controller |     - '        '
2026-03-14 09:56:26.773824 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.773828 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.773832 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.773837 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.773841 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.773845 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.773850 | controller |       -- \'
2026-03-14 09:56:26.773854 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.773858 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.773863 | controller |     - '    logger.go:42: 09:46:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.773867 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.773871 | controller |     - '    logger.go:42: 09:46:24 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.773876 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.773880 | controller |       found'
2026-03-14 09:56:26.773884 | controller |     - '    logger.go:42: 09:46:25 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.773889 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.773893 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.773897 | controller |     - '        '
2026-03-14 09:56:26.773902 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.773906 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.773910 | controller |     - '        '
2026-03-14 09:56:26.773915 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.773919 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.773923 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.773928 | controller |     - '          exit 1'
2026-03-14 09:56:26.773932 | controller |     - '        fi'
2026-03-14 09:56:26.773936 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.773940 | controller |     - '        '
2026-03-14 09:56:26.773945 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.773949 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.773953 | controller |       | base64 -d)'
2026-03-14 09:56:26.773958 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.773962 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.773966 | controller |     - '          exit 1'
2026-03-14 09:56:26.773971 | controller |     - '        fi'
2026-03-14 09:56:26.773975 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.773979 | controller |     - '        '
2026-03-14 09:56:26.773984 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.773989 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.773994 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.773998 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.774002 | controller |       --timeout=300s'
2026-03-14 09:56:26.774007 | controller |     - '        '
2026-03-14 09:56:26.774011 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.774015 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.774022 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.774026 | controller |     - '        '
2026-03-14 09:56:26.774031 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.774035 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.774046 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.774051 | controller |     - '        '
2026-03-14 09:56:26.774055 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.774059 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.774064 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.774068 | controller |       available"'
2026-03-14 09:56:26.774072 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.774077 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.774084 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.774088 | controller |     - '          exit 1'
2026-03-14 09:56:26.774093 | controller |     - '        fi'
2026-03-14 09:56:26.774164 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.774170 | controller |     - '        '
2026-03-14 09:56:26.774175 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.774179 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.774184 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.774188 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.774193 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.774197 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.774201 | controller |       -- \'
2026-03-14 09:56:26.774206 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.774210 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.774215 | controller |     - '    logger.go:42: 09:46:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.774219 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.774223 | controller |     - '    logger.go:42: 09:46:26 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.774228 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.774232 | controller |       found'
2026-03-14 09:56:26.774236 | controller |     - '    logger.go:42: 09:46:27 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.774243 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.774249 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.774254 | controller |     - '        '
2026-03-14 09:56:26.774258 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.774263 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.774267 | controller |     - '        '
2026-03-14 09:56:26.774271 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.774276 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.774280 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.774285 | controller |     - '          exit 1'
2026-03-14 09:56:26.774289 | controller |     - '        fi'
2026-03-14 09:56:26.774293 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.774298 | controller |     - '        '
2026-03-14 09:56:26.774302 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.774306 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.774311 | controller |       | base64 -d)'
2026-03-14 09:56:26.774315 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.774319 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.774324 | controller |     - '          exit 1'
2026-03-14 09:56:26.774328 | controller |     - '        fi'
2026-03-14 09:56:26.774333 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.774337 | controller |     - '        '
2026-03-14 09:56:26.774341 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.774346 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.774350 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.774354 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.774359 | controller |       --timeout=300s'
2026-03-14 09:56:26.774363 | controller |     - '        '
2026-03-14 09:56:26.774367 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.774372 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.774376 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.774380 | controller |     - '        '
2026-03-14 09:56:26.774385 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.774389 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.774402 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.774406 | controller |     - '        '
2026-03-14 09:56:26.774411 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.774415 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.774420 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.774424 | controller |       available"'
2026-03-14 09:56:26.774428 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.774433 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.774437 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.774441 | controller |     - '          exit 1'
2026-03-14 09:56:26.774446 | controller |     - '        fi'
2026-03-14 09:56:26.774450 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.774455 | controller |     - '        '
2026-03-14 09:56:26.774461 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.774465 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.774469 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.774474 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.774478 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.774483 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.774487 | controller |       -- \'
2026-03-14 09:56:26.774491 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.774496 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.774500 | controller |     - '    logger.go:42: 09:46:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.774504 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.774509 | controller |     - '    logger.go:42: 09:46:27 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.774513 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.774517 | controller |       found'
2026-03-14 09:56:26.774522 | controller |     - '    logger.go:42: 09:46:28 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.774526 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.774530 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.774535 | controller |     - '        '
2026-03-14 09:56:26.774541 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.774546 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.774550 | controller |     - '        '
2026-03-14 09:56:26.774554 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.774559 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.774563 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.774567 | controller |     - '          exit 1'
2026-03-14 09:56:26.774572 | controller |     - '        fi'
2026-03-14 09:56:26.774576 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.774580 | controller |     - '        '
2026-03-14 09:56:26.774585 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.774589 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.774605 | controller |       | base64 -d)'
2026-03-14 09:56:26.774624 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.774629 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.774633 | controller |     - '          exit 1'
2026-03-14 09:56:26.774638 | controller |     - '        fi'
2026-03-14 09:56:26.774642 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.774646 | controller |     - '        '
2026-03-14 09:56:26.774651 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.774655 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.774659 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.774664 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.774668 | controller |       --timeout=300s'
2026-03-14 09:56:26.774673 | controller |     - '        '
2026-03-14 09:56:26.774677 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.774683 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.774688 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.774692 | controller |     - '        '
2026-03-14 09:56:26.774697 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.774701 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.774706 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.774712 | controller |     - '        '
2026-03-14 09:56:26.774724 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.774729 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.774734 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.774738 | controller |       available"'
2026-03-14 09:56:26.774742 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.774747 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.774751 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.774756 | controller |     - '          exit 1'
2026-03-14 09:56:26.774760 | controller |     - '        fi'
2026-03-14 09:56:26.774764 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.774769 | controller |     - '        '
2026-03-14 09:56:26.774773 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.774777 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.774782 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.774786 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.774791 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.774795 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.774799 | controller |       -- \'
2026-03-14 09:56:26.774804 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.774808 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.774813 | controller |     - '    logger.go:42: 09:46:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.774817 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.774821 | controller |     - '    logger.go:42: 09:46:28 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.774826 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.774830 | controller |       found'
2026-03-14 09:56:26.774834 | controller |     - '    logger.go:42: 09:46:29 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.774839 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.774843 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.774847 | controller |     - '        '
2026-03-14 09:56:26.774852 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.774858 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.774862 | controller |     - '        '
2026-03-14 09:56:26.774867 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.774871 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.774877 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.774881 | controller |     - '          exit 1'
2026-03-14 09:56:26.774886 | controller |     - '        fi'
2026-03-14 09:56:26.774890 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.774894 | controller |     - '        '
2026-03-14 09:56:26.774899 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.774903 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.774907 | controller |       | base64 -d)'
2026-03-14 09:56:26.774912 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.774916 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.774921 | controller |     - '          exit 1'
2026-03-14 09:56:26.774925 | controller |     - '        fi'
2026-03-14 09:56:26.774929 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.774934 | controller |     - '        '
2026-03-14 09:56:26.774938 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.774942 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.774947 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.774951 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.774956 | controller |       --timeout=300s'
2026-03-14 09:56:26.774960 | controller |     - '        '
2026-03-14 09:56:26.774964 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.774969 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.774973 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.774977 | controller |     - '        '
2026-03-14 09:56:26.774982 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.774986 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.774990 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.774995 | controller |     - '        '
2026-03-14 09:56:26.774999 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.775010 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.775015 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.775019 | controller |       available"'
2026-03-14 09:56:26.775042 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.775047 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.775051 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.775056 | controller |     - '          exit 1'
2026-03-14 09:56:26.775060 | controller |     - '        fi'
2026-03-14 09:56:26.775064 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.775069 | controller |     - '        '
2026-03-14 09:56:26.775073 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.775078 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.775082 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.775089 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.775093 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.775099 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.775105 | controller |       -- \'
2026-03-14 09:56:26.775110 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.775114 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.775119 | controller |     - '    logger.go:42: 09:46:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.775123 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.775127 | controller |     - '    logger.go:42: 09:46:29 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.775132 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.775136 | controller |       found'
2026-03-14 09:56:26.775141 | controller |     - '    logger.go:42: 09:46:30 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.775145 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.775149 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.775154 | controller |     - '        '
2026-03-14 09:56:26.775158 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.775162 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.775167 | controller |     - '        '
2026-03-14 09:56:26.775171 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.775175 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.775180 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.775184 | controller |     - '          exit 1'
2026-03-14 09:56:26.775188 | controller |     - '        fi'
2026-03-14 09:56:26.775193 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.775197 | controller |     - '        '
2026-03-14 09:56:26.775201 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.775206 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.775210 | controller |       | base64 -d)'
2026-03-14 09:56:26.775215 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.775219 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.775223 | controller |     - '          exit 1'
2026-03-14 09:56:26.775228 | controller |     - '        fi'
2026-03-14 09:56:26.775232 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.775236 | controller |     - '        '
2026-03-14 09:56:26.775241 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.775245 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.775249 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.775254 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.775258 | controller |       --timeout=300s'
2026-03-14 09:56:26.775262 | controller |     - '        '
2026-03-14 09:56:26.775267 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.775271 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.775275 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.775280 | controller |     - '        '
2026-03-14 09:56:26.775284 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.775289 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.775294 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.775299 | controller |     - '        '
2026-03-14 09:56:26.775303 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.775315 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.775319 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.775324 | controller |       available"'
2026-03-14 09:56:26.775328 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.775332 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.775337 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.775341 | controller |     - '          exit 1'
2026-03-14 09:56:26.775345 | controller |     - '        fi'
2026-03-14 09:56:26.775349 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.775354 | controller |     - '        '
2026-03-14 09:56:26.775358 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.775364 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.775370 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.775376 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.775382 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.775387 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.775391 | controller |       -- \'
2026-03-14 09:56:26.775396 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.775402 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.775408 | controller |     - '    logger.go:42: 09:46:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.775413 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.775418 | controller |     - '    logger.go:42: 09:46:30 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.775423 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.775429 | controller |       found'
2026-03-14 09:56:26.775435 | controller |     - '    logger.go:42: 09:46:31 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.775440 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.775444 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.775448 | controller |     - '        '
2026-03-14 09:56:26.775453 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.775459 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.775468 | controller |     - '        '
2026-03-14 09:56:26.775472 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.775477 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.775482 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.775488 | controller |     - '          exit 1'
2026-03-14 09:56:26.775494 | controller |     - '        fi'
2026-03-14 09:56:26.775499 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.775503 | controller |     - '        '
2026-03-14 09:56:26.775507 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.775513 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.775518 | controller |       | base64 -d)'
2026-03-14 09:56:26.775525 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.775530 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.775534 | controller |     - '          exit 1'
2026-03-14 09:56:26.775538 | controller |     - '        fi'
2026-03-14 09:56:26.775543 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.775547 | controller |     - '        '
2026-03-14 09:56:26.775551 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.775556 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.775560 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.775564 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.775569 | controller |       --timeout=300s'
2026-03-14 09:56:26.775573 | controller |     - '        '
2026-03-14 09:56:26.775577 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.775582 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.775586 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.775590 | controller |     - '        '
2026-03-14 09:56:26.775595 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.775616 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.775622 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.775628 | controller |     - '        '
2026-03-14 09:56:26.775634 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.775640 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.775655 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.775662 | controller |       available"'
2026-03-14 09:56:26.775667 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.775672 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.775676 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.775682 | controller |     - '          exit 1'
2026-03-14 09:56:26.775688 | controller |     - '        fi'
2026-03-14 09:56:26.775693 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.775698 | controller |     - '        '
2026-03-14 09:56:26.775702 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.775706 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.775711 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.775715 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.775719 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.775724 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.775728 | controller |       -- \'
2026-03-14 09:56:26.775732 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.775737 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.775741 | controller |     - '    logger.go:42: 09:46:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.775748 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.775752 | controller |     - '    logger.go:42: 09:46:31 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.775756 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.775761 | controller |       found'
2026-03-14 09:56:26.775765 | controller |     - '    logger.go:42: 09:46:32 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.775769 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.775774 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.775778 | controller |     - '        '
2026-03-14 09:56:26.775782 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.775787 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.775791 | controller |     - '        '
2026-03-14 09:56:26.775795 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.775800 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.775806 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.775811 | controller |     - '          exit 1'
2026-03-14 09:56:26.775815 | controller |     - '        fi'
2026-03-14 09:56:26.775820 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.775825 | controller |     - '        '
2026-03-14 09:56:26.775831 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.775837 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.775842 | controller |       | base64 -d)'
2026-03-14 09:56:26.775847 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.775851 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.775857 | controller |     - '          exit 1'
2026-03-14 09:56:26.775863 | controller |     - '        fi'
2026-03-14 09:56:26.775870 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.775875 | controller |     - '        '
2026-03-14 09:56:26.775879 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.775885 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.775891 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.775896 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.775900 | controller |       --timeout=300s'
2026-03-14 09:56:26.775905 | controller |     - '        '
2026-03-14 09:56:26.775911 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.775917 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.775923 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.775927 | controller |     - '        '
2026-03-14 09:56:26.775931 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.775936 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.775940 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.775944 | controller |     - '        '
2026-03-14 09:56:26.775949 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.775953 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.775965 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.775970 | controller |       available"'
2026-03-14 09:56:26.775975 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.775982 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.775986 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.775991 | controller |     - '          exit 1'
2026-03-14 09:56:26.775995 | controller |     - '        fi'
2026-03-14 09:56:26.775999 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.776004 | controller |     - '        '
2026-03-14 09:56:26.776008 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.776012 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.776017 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.776021 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.776026 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.776030 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.776034 | controller |       -- \'
2026-03-14 09:56:26.776039 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.776043 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.776047 | controller |     - '    logger.go:42: 09:46:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.776052 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.776058 | controller |     - '    logger.go:42: 09:46:33 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.776064 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.776070 | controller |       found'
2026-03-14 09:56:26.776075 | controller |     - '    logger.go:42: 09:46:34 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.776081 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.776087 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.776092 | controller |     - '        '
2026-03-14 09:56:26.776097 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.776103 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.776109 | controller |     - '        '
2026-03-14 09:56:26.776114 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.776121 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.776127 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.776133 | controller |     - '          exit 1'
2026-03-14 09:56:26.776139 | controller |     - '        fi'
2026-03-14 09:56:26.776143 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.776148 | controller |     - '        '
2026-03-14 09:56:26.776152 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.776156 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.776160 | controller |       | base64 -d)'
2026-03-14 09:56:26.776165 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.776169 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.776173 | controller |     - '          exit 1'
2026-03-14 09:56:26.776178 | controller |     - '        fi'
2026-03-14 09:56:26.776215 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.776222 | controller |     - '        '
2026-03-14 09:56:26.776227 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.776231 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.776238 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.776242 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.776246 | controller |       --timeout=300s'
2026-03-14 09:56:26.776251 | controller |     - '        '
2026-03-14 09:56:26.776255 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.776259 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.776264 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.776270 | controller |     - '        '
2026-03-14 09:56:26.776275 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.776280 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.776285 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.776291 | controller |     - '        '
2026-03-14 09:56:26.776297 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.776303 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.776309 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.776315 | controller |       available"'
2026-03-14 09:56:26.776332 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.776339 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.776345 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.776350 | controller |     - '          exit 1'
2026-03-14 09:56:26.776357 | controller |     - '        fi'
2026-03-14 09:56:26.776364 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.776370 | controller |     - '        '
2026-03-14 09:56:26.776374 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.776379 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.776383 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.776387 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.776392 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.776396 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.776400 | controller |       -- \'
2026-03-14 09:56:26.776404 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.776409 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.776413 | controller |     - '    logger.go:42: 09:46:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.776417 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.776422 | controller |     - '    logger.go:42: 09:46:34 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.776426 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.776430 | controller |       found'
2026-03-14 09:56:26.776435 | controller |     - '    logger.go:42: 09:46:35 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.776439 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.776443 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.776450 | controller |     - '        '
2026-03-14 09:56:26.776454 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.776458 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.776463 | controller |     - '        '
2026-03-14 09:56:26.776467 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.776471 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.776476 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.776480 | controller |     - '          exit 1'
2026-03-14 09:56:26.776484 | controller |     - '        fi'
2026-03-14 09:56:26.776488 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.776493 | controller |     - '        '
2026-03-14 09:56:26.776498 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.776504 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.776509 | controller |       | base64 -d)'
2026-03-14 09:56:26.776515 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.776521 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.776527 | controller |     - '          exit 1'
2026-03-14 09:56:26.776533 | controller |     - '        fi'
2026-03-14 09:56:26.776539 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.776544 | controller |     - '        '
2026-03-14 09:56:26.776548 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.776553 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.776557 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.776561 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.776566 | controller |       --timeout=300s'
2026-03-14 09:56:26.776570 | controller |     - '        '
2026-03-14 09:56:26.776574 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.776581 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.776586 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.776590 | controller |     - '        '
2026-03-14 09:56:26.776607 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.776623 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.776628 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.776632 | controller |     - '        '
2026-03-14 09:56:26.776637 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.776641 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.776645 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.776650 | controller |       available"'
2026-03-14 09:56:26.776654 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.776667 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.776671 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.776676 | controller |     - '          exit 1'
2026-03-14 09:56:26.776680 | controller |     - '        fi'
2026-03-14 09:56:26.776684 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.776689 | controller |     - '        '
2026-03-14 09:56:26.776693 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.776700 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.776704 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.776708 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.776713 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.776717 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.776721 | controller |       -- \'
2026-03-14 09:56:26.776726 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.776730 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.776734 | controller |     - '    logger.go:42: 09:46:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.776738 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.776743 | controller |     - '    logger.go:42: 09:46:35 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.776747 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.776751 | controller |       found'
2026-03-14 09:56:26.776756 | controller |     - '    logger.go:42: 09:46:36 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.776760 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.776764 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.776769 | controller |     - '        '
2026-03-14 09:56:26.776773 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.776777 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.776781 | controller |     - '        '
2026-03-14 09:56:26.776786 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.776790 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.776794 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.776799 | controller |     - '          exit 1'
2026-03-14 09:56:26.776803 | controller |     - '        fi'
2026-03-14 09:56:26.776807 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.776812 | controller |     - '        '
2026-03-14 09:56:26.776816 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.776820 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.776825 | controller |       | base64 -d)'
2026-03-14 09:56:26.776829 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.776833 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.776838 | controller |     - '          exit 1'
2026-03-14 09:56:26.776842 | controller |     - '        fi'
2026-03-14 09:56:26.776846 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.776851 | controller |     - '        '
2026-03-14 09:56:26.776855 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.776859 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.776866 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.776870 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.776874 | controller |       --timeout=300s'
2026-03-14 09:56:26.776878 | controller |     - '        '
2026-03-14 09:56:26.776883 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.776887 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.776893 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.776897 | controller |     - '        '
2026-03-14 09:56:26.776902 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.776906 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.776910 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.776915 | controller |     - '        '
2026-03-14 09:56:26.776919 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.776923 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.776928 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.776932 | controller |       available"'
2026-03-14 09:56:26.776936 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.776949 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.776953 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.776958 | controller |     - '          exit 1'
2026-03-14 09:56:26.776962 | controller |     - '        fi'
2026-03-14 09:56:26.776966 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.776971 | controller |     - '        '
2026-03-14 09:56:26.776975 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.776979 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.776984 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.776988 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.776992 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.776997 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.777001 | controller |       -- \'
2026-03-14 09:56:26.777005 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.777009 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.777014 | controller |     - '    logger.go:42: 09:46:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.777018 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.777022 | controller |     - '    logger.go:42: 09:46:36 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.777027 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.777031 | controller |       found'
2026-03-14 09:56:26.777035 | controller |     - '    logger.go:42: 09:46:37 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.777039 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.777044 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.777048 | controller |     - '        '
2026-03-14 09:56:26.777052 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.777056 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.777061 | controller |     - '        '
2026-03-14 09:56:26.777065 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.777069 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.777074 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.777081 | controller |     - '          exit 1'
2026-03-14 09:56:26.777089 | controller |     - '        fi'
2026-03-14 09:56:26.777093 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.777098 | controller |     - '        '
2026-03-14 09:56:26.777102 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.777106 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.777111 | controller |       | base64 -d)'
2026-03-14 09:56:26.777115 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.777119 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.777124 | controller |     - '          exit 1'
2026-03-14 09:56:26.777130 | controller |     - '        fi'
2026-03-14 09:56:26.777134 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.777139 | controller |     - '        '
2026-03-14 09:56:26.777143 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.777148 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.777152 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.777156 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.777160 | controller |       --timeout=300s'
2026-03-14 09:56:26.777165 | controller |     - '        '
2026-03-14 09:56:26.777169 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.777173 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.777178 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.777182 | controller |     - '        '
2026-03-14 09:56:26.777186 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.777191 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.777195 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.777199 | controller |     - '        '
2026-03-14 09:56:26.777204 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.777208 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.777212 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.777217 | controller |       available"'
2026-03-14 09:56:26.777221 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.777225 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.777236 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.777240 | controller |     - '          exit 1'
2026-03-14 09:56:26.777245 | controller |     - '        fi'
2026-03-14 09:56:26.777249 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.777253 | controller |     - '        '
2026-03-14 09:56:26.777258 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.777262 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.777266 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.777271 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.777275 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.777281 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.777287 | controller |       -- \'
2026-03-14 09:56:26.777291 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.777296 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.777300 | controller |     - '    logger.go:42: 09:46:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.777304 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.777309 | controller |     - '    logger.go:42: 09:46:37 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.777313 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.777317 | controller |       found'
2026-03-14 09:56:26.777322 | controller |     - '    logger.go:42: 09:46:38 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.777326 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.777330 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.777334 | controller |     - '        '
2026-03-14 09:56:26.777339 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.777343 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.777347 | controller |     - '        '
2026-03-14 09:56:26.777352 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.777356 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.777360 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.777365 | controller |     - '          exit 1'
2026-03-14 09:56:26.777369 | controller |     - '        fi'
2026-03-14 09:56:26.777373 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.777378 | controller |     - '        '
2026-03-14 09:56:26.777382 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.777386 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.777391 | controller |       | base64 -d)'
2026-03-14 09:56:26.777395 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.777399 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.777404 | controller |     - '          exit 1'
2026-03-14 09:56:26.777408 | controller |     - '        fi'
2026-03-14 09:56:26.777412 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.777416 | controller |     - '        '
2026-03-14 09:56:26.777421 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.777425 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.777429 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.777436 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.777440 | controller |       --timeout=300s'
2026-03-14 09:56:26.777444 | controller |     - '        '
2026-03-14 09:56:26.777449 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.777453 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.777457 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.777462 | controller |     - '        '
2026-03-14 09:56:26.777466 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.777470 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.777475 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.777481 | controller |     - '        '
2026-03-14 09:56:26.777485 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.777489 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.777494 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.777498 | controller |       available"'
2026-03-14 09:56:26.777502 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.777507 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.777511 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.777521 | controller |     - '          exit 1'
2026-03-14 09:56:26.777526 | controller |     - '        fi'
2026-03-14 09:56:26.777530 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.777535 | controller |     - '        '
2026-03-14 09:56:26.777539 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.777543 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.777547 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.777552 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.777556 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.777560 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.777565 | controller |       -- \'
2026-03-14 09:56:26.777569 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.777573 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.777578 | controller |     - '    logger.go:42: 09:46:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.777582 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.777586 | controller |     - '    logger.go:42: 09:46:38 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.777591 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.777609 | controller |       found'
2026-03-14 09:56:26.777613 | controller |     - '    logger.go:42: 09:46:39 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.777618 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.777622 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.777626 | controller |     - '        '
2026-03-14 09:56:26.777631 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.777635 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.777639 | controller |     - '        '
2026-03-14 09:56:26.777643 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.777648 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.777652 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.777656 | controller |     - '          exit 1'
2026-03-14 09:56:26.777661 | controller |     - '        fi'
2026-03-14 09:56:26.777665 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.777669 | controller |     - '        '
2026-03-14 09:56:26.777674 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.777678 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.777682 | controller |       | base64 -d)'
2026-03-14 09:56:26.777687 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.777693 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.777697 | controller |     - '          exit 1'
2026-03-14 09:56:26.777702 | controller |     - '        fi'
2026-03-14 09:56:26.777706 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.777710 | controller |     - '        '
2026-03-14 09:56:26.777715 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.777719 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.777723 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.777728 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.777732 | controller |       --timeout=300s'
2026-03-14 09:56:26.777736 | controller |     - '        '
2026-03-14 09:56:26.777741 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.777745 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.777749 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.777754 | controller |     - '        '
2026-03-14 09:56:26.777758 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.777762 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.777767 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.777771 | controller |     - '        '
2026-03-14 09:56:26.777775 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.777780 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.777784 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.777788 | controller |       available"'
2026-03-14 09:56:26.777793 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.777797 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.777801 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.777805 | controller |     - '          exit 1'
2026-03-14 09:56:26.777810 | controller |     - '        fi'
2026-03-14 09:56:26.777821 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.777825 | controller |     - '        '
2026-03-14 09:56:26.777830 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.777834 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.777838 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.777843 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.777847 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.777851 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.777855 | controller |       -- \'
2026-03-14 09:56:26.777860 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.777864 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.777869 | controller |     - '    logger.go:42: 09:46:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.777873 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.777879 | controller |     - '    logger.go:42: 09:46:40 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.777883 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.777888 | controller |       found'
2026-03-14 09:56:26.777892 | controller |     - '    logger.go:42: 09:46:41 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.777896 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.777901 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.777905 | controller |     - '        '
2026-03-14 09:56:26.777909 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.777914 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.777918 | controller |     - '        '
2026-03-14 09:56:26.777922 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.777927 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.777931 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.777935 | controller |     - '          exit 1'
2026-03-14 09:56:26.777940 | controller |     - '        fi'
2026-03-14 09:56:26.777944 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.777950 | controller |     - '        '
2026-03-14 09:56:26.777955 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.777959 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.777965 | controller |       | base64 -d)'
2026-03-14 09:56:26.777969 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.777973 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.777978 | controller |     - '          exit 1'
2026-03-14 09:56:26.777982 | controller |     - '        fi'
2026-03-14 09:56:26.777986 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.777991 | controller |     - '        '
2026-03-14 09:56:26.777995 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.777999 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.778004 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.778008 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.778014 | controller |       --timeout=300s'
2026-03-14 09:56:26.778018 | controller |     - '        '
2026-03-14 09:56:26.778022 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.778027 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.778031 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.778035 | controller |     - '        '
2026-03-14 09:56:26.778040 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.778044 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.778048 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.778052 | controller |     - '        '
2026-03-14 09:56:26.778057 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.778061 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.778065 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.778070 | controller |       available"'
2026-03-14 09:56:26.778074 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.778080 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.778084 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.778089 | controller |     - '          exit 1'
2026-03-14 09:56:26.778095 | controller |     - '        fi'
2026-03-14 09:56:26.778105 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.778110 | controller |     - '        '
2026-03-14 09:56:26.778114 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.778118 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.778123 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.778127 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.778131 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.778136 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.778140 | controller |       -- \'
2026-03-14 09:56:26.778144 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.778149 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.778153 | controller |     - '    logger.go:42: 09:46:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.778157 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.778161 | controller |     - '    logger.go:42: 09:46:41 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.778166 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.778170 | controller |       found'
2026-03-14 09:56:26.778174 | controller |     - '    logger.go:42: 09:46:42 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.778178 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.778183 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.778187 | controller |     - '        '
2026-03-14 09:56:26.778191 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.778196 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.778200 | controller |     - '        '
2026-03-14 09:56:26.778204 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.778208 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.778213 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.778217 | controller |     - '          exit 1'
2026-03-14 09:56:26.778221 | controller |     - '        fi'
2026-03-14 09:56:26.778226 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.778230 | controller |     - '        '
2026-03-14 09:56:26.778234 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.778240 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.778245 | controller |       | base64 -d)'
2026-03-14 09:56:26.778249 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.778253 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.778257 | controller |     - '          exit 1'
2026-03-14 09:56:26.778262 | controller |     - '        fi'
2026-03-14 09:56:26.778266 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.778270 | controller |     - '        '
2026-03-14 09:56:26.778275 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.778279 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.778283 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.778289 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.778293 | controller |       --timeout=300s'
2026-03-14 09:56:26.778298 | controller |     - '        '
2026-03-14 09:56:26.778302 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.778306 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.778311 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.778315 | controller |     - '        '
2026-03-14 09:56:26.778319 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.778324 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.778332 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.778336 | controller |     - '        '
2026-03-14 09:56:26.778341 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.778345 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.778349 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.778354 | controller |       available"'
2026-03-14 09:56:26.778358 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.778362 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.778367 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.778371 | controller |     - '          exit 1'
2026-03-14 09:56:26.778375 | controller |     - '        fi'
2026-03-14 09:56:26.778380 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.778384 | controller |     - '        '
2026-03-14 09:56:26.778395 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.778399 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.778404 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.778408 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.778412 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.778416 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.778421 | controller |       -- \'
2026-03-14 09:56:26.778425 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.778429 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.778434 | controller |     - '    logger.go:42: 09:46:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.778438 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.778442 | controller |     - '    logger.go:42: 09:46:42 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.778447 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.778451 | controller |       found'
2026-03-14 09:56:26.778455 | controller |     - '    logger.go:42: 09:46:43 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.778459 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.778464 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.778468 | controller |     - '        '
2026-03-14 09:56:26.778472 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.778478 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.778483 | controller |     - '        '
2026-03-14 09:56:26.778487 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.778491 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.778496 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.778500 | controller |     - '          exit 1'
2026-03-14 09:56:26.778504 | controller |     - '        fi'
2026-03-14 09:56:26.778509 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.778513 | controller |     - '        '
2026-03-14 09:56:26.778517 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.778522 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.778526 | controller |       | base64 -d)'
2026-03-14 09:56:26.778530 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.778535 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.778539 | controller |     - '          exit 1'
2026-03-14 09:56:26.778543 | controller |     - '        fi'
2026-03-14 09:56:26.778547 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.778552 | controller |     - '        '
2026-03-14 09:56:26.778556 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.778560 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.778565 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.778569 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.778573 | controller |       --timeout=300s'
2026-03-14 09:56:26.778578 | controller |     - '        '
2026-03-14 09:56:26.778582 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.778586 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.778590 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.778608 | controller |     - '        '
2026-03-14 09:56:26.778621 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.778625 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.778629 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.778634 | controller |     - '        '
2026-03-14 09:56:26.778638 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.778645 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.778649 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.778653 | controller |       available"'
2026-03-14 09:56:26.778658 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.778662 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.778666 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.778670 | controller |     - '          exit 1'
2026-03-14 09:56:26.778675 | controller |     - '        fi'
2026-03-14 09:56:26.778679 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.778683 | controller |     - '        '
2026-03-14 09:56:26.778688 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.778702 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.778708 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.778712 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.778716 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.778721 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.778725 | controller |       -- \'
2026-03-14 09:56:26.778729 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.778734 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.778738 | controller |     - '    logger.go:42: 09:46:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.778742 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.778747 | controller |     - '    logger.go:42: 09:46:43 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.778751 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.778755 | controller |       found'
2026-03-14 09:56:26.778759 | controller |     - '    logger.go:42: 09:46:44 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.778764 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.778768 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.778772 | controller |     - '        '
2026-03-14 09:56:26.778777 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.778781 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.778785 | controller |     - '        '
2026-03-14 09:56:26.778789 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.778794 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.778798 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.778802 | controller |     - '          exit 1'
2026-03-14 09:56:26.778807 | controller |     - '        fi'
2026-03-14 09:56:26.778811 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.778815 | controller |     - '        '
2026-03-14 09:56:26.778819 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.778824 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.778828 | controller |       | base64 -d)'
2026-03-14 09:56:26.778832 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.778837 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.778841 | controller |     - '          exit 1'
2026-03-14 09:56:26.778845 | controller |     - '        fi'
2026-03-14 09:56:26.778849 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.778854 | controller |     - '        '
2026-03-14 09:56:26.778858 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.778862 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.778867 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.778871 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.778875 | controller |       --timeout=300s'
2026-03-14 09:56:26.778879 | controller |     - '        '
2026-03-14 09:56:26.778884 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.778888 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.778894 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.778899 | controller |     - '        '
2026-03-14 09:56:26.778903 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.778907 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.778912 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.778916 | controller |     - '        '
2026-03-14 09:56:26.778920 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.778925 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.778929 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.778933 | controller |       available"'
2026-03-14 09:56:26.778937 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.778942 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.778946 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.778950 | controller |     - '          exit 1'
2026-03-14 09:56:26.778955 | controller |     - '        fi'
2026-03-14 09:56:26.778959 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.778963 | controller |     - '        '
2026-03-14 09:56:26.778968 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.778978 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.778982 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.778987 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.778991 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.778995 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.779000 | controller |       -- \'
2026-03-14 09:56:26.779004 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.779008 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.779013 | controller |     - '    logger.go:42: 09:46:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.779017 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.779022 | controller |     - '    logger.go:42: 09:46:44 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.779042 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.779047 | controller |       found'
2026-03-14 09:56:26.779051 | controller |     - '    logger.go:42: 09:46:45 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.779055 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.779060 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.779064 | controller |     - '        '
2026-03-14 09:56:26.779068 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.779073 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.779077 | controller |     - '        '
2026-03-14 09:56:26.779081 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.779085 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.779090 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.779094 | controller |     - '          exit 1'
2026-03-14 09:56:26.779100 | controller |     - '        fi'
2026-03-14 09:56:26.779105 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.779109 | controller |     - '        '
2026-03-14 09:56:26.779113 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.779118 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.779122 | controller |       | base64 -d)'
2026-03-14 09:56:26.779126 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.779131 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.779135 | controller |     - '          exit 1'
2026-03-14 09:56:26.779139 | controller |     - '        fi'
2026-03-14 09:56:26.779144 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.779148 | controller |     - '        '
2026-03-14 09:56:26.779152 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.779157 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.779161 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.779165 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.779170 | controller |       --timeout=300s'
2026-03-14 09:56:26.779176 | controller |     - '        '
2026-03-14 09:56:26.779180 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.779185 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.779189 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.779193 | controller |     - '        '
2026-03-14 09:56:26.779198 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.779202 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.779206 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.779211 | controller |     - '        '
2026-03-14 09:56:26.779215 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.779219 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.779224 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.779229 | controller |       available"'
2026-03-14 09:56:26.779234 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.779238 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.779242 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.779247 | controller |     - '          exit 1'
2026-03-14 09:56:26.779251 | controller |     - '        fi'
2026-03-14 09:56:26.779255 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.779260 | controller |     - '        '
2026-03-14 09:56:26.779264 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.779268 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.779279 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.779284 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.779289 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.779293 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.779301 | controller |       -- \'
2026-03-14 09:56:26.779305 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.779311 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.779315 | controller |     - '    logger.go:42: 09:46:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.779320 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.779324 | controller |     - '    logger.go:42: 09:46:45 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.779328 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.779333 | controller |       found'
2026-03-14 09:56:26.779337 | controller |     - '    logger.go:42: 09:46:46 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.779342 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.779346 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.779350 | controller |     - '        '
2026-03-14 09:56:26.779355 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.779361 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.779366 | controller |     - '        '
2026-03-14 09:56:26.779370 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.779375 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.779381 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.779385 | controller |     - '          exit 1'
2026-03-14 09:56:26.779390 | controller |     - '        fi'
2026-03-14 09:56:26.779394 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.779398 | controller |     - '        '
2026-03-14 09:56:26.779402 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.779407 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.779411 | controller |       | base64 -d)'
2026-03-14 09:56:26.779415 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.779419 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.779424 | controller |     - '          exit 1'
2026-03-14 09:56:26.779428 | controller |     - '        fi'
2026-03-14 09:56:26.779432 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.779437 | controller |     - '        '
2026-03-14 09:56:26.779441 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.779445 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.779449 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.779454 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.779458 | controller |       --timeout=300s'
2026-03-14 09:56:26.779462 | controller |     - '        '
2026-03-14 09:56:26.779467 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.779471 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.779475 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.779480 | controller |     - '        '
2026-03-14 09:56:26.779484 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.779488 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.779492 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.779500 | controller |     - '        '
2026-03-14 09:56:26.779504 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.779509 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.779513 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.779517 | controller |       available"'
2026-03-14 09:56:26.779522 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.779526 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.779532 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.779536 | controller |     - '          exit 1'
2026-03-14 09:56:26.779540 | controller |     - '        fi'
2026-03-14 09:56:26.779545 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.779549 | controller |     - '        '
2026-03-14 09:56:26.779553 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.779558 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.779568 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.779573 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.779577 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.779581 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.779586 | controller |       -- \'
2026-03-14 09:56:26.779590 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.779605 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.779612 | controller |     - '    logger.go:42: 09:46:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.779616 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.779621 | controller |     - '    logger.go:42: 09:46:47 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.779625 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.779629 | controller |       found'
2026-03-14 09:56:26.779634 | controller |     - '    logger.go:42: 09:46:48 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.779638 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.779652 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.779657 | controller |     - '        '
2026-03-14 09:56:26.779661 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.779665 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.779670 | controller |     - '        '
2026-03-14 09:56:26.779674 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.779678 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.779683 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.779687 | controller |     - '          exit 1'
2026-03-14 09:56:26.779692 | controller |     - '        fi'
2026-03-14 09:56:26.779696 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.779700 | controller |     - '        '
2026-03-14 09:56:26.779704 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.779709 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.779713 | controller |       | base64 -d)'
2026-03-14 09:56:26.779717 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.779722 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.779729 | controller |     - '          exit 1'
2026-03-14 09:56:26.779733 | controller |     - '        fi'
2026-03-14 09:56:26.779738 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.779742 | controller |     - '        '
2026-03-14 09:56:26.779746 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.779751 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.779755 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.779759 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.779764 | controller |       --timeout=300s'
2026-03-14 09:56:26.779768 | controller |     - '        '
2026-03-14 09:56:26.779772 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.779777 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.779781 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.779785 | controller |     - '        '
2026-03-14 09:56:26.779790 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.779794 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.779798 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.779803 | controller |     - '        '
2026-03-14 09:56:26.779807 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.779811 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.779815 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.779820 | controller |       available"'
2026-03-14 09:56:26.779824 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.779828 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.779833 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.779837 | controller |     - '          exit 1'
2026-03-14 09:56:26.779841 | controller |     - '        fi'
2026-03-14 09:56:26.779846 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.779851 | controller |     - '        '
2026-03-14 09:56:26.779856 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.779860 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.779864 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.779876 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.779880 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.779885 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.779889 | controller |       -- \'
2026-03-14 09:56:26.779894 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.779898 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.779902 | controller |     - '    logger.go:42: 09:46:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.779906 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.779911 | controller |     - '    logger.go:42: 09:46:48 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.779917 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.779921 | controller |       found'
2026-03-14 09:56:26.779925 | controller |     - '    logger.go:42: 09:46:49 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.779930 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.779934 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.779938 | controller |     - '        '
2026-03-14 09:56:26.779943 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.779947 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.779951 | controller |     - '        '
2026-03-14 09:56:26.779956 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.779960 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.779964 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.779968 | controller |     - '          exit 1'
2026-03-14 09:56:26.779973 | controller |     - '        fi'
2026-03-14 09:56:26.779977 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.779981 | controller |     - '        '
2026-03-14 09:56:26.779986 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.779990 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.779994 | controller |       | base64 -d)'
2026-03-14 09:56:26.779999 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.780003 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.780007 | controller |     - '          exit 1'
2026-03-14 09:56:26.780012 | controller |     - '        fi'
2026-03-14 09:56:26.780016 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.780020 | controller |     - '        '
2026-03-14 09:56:26.780025 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.780029 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.780033 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.780038 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.780042 | controller |       --timeout=300s'
2026-03-14 09:56:26.780046 | controller |     - '        '
2026-03-14 09:56:26.780050 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.780055 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.780059 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.780063 | controller |     - '        '
2026-03-14 09:56:26.780068 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.780072 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.780076 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.780081 | controller |     - '        '
2026-03-14 09:56:26.780085 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.780089 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.780093 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.780098 | controller |       available"'
2026-03-14 09:56:26.780102 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.780106 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.780112 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.780117 | controller |     - '          exit 1'
2026-03-14 09:56:26.780121 | controller |     - '        fi'
2026-03-14 09:56:26.780125 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.780130 | controller |     - '        '
2026-03-14 09:56:26.780134 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.780138 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.780142 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.780153 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.780157 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.780162 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.780166 | controller |       -- \'
2026-03-14 09:56:26.780170 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.780175 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.780179 | controller |     - '    logger.go:42: 09:46:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.780183 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.780188 | controller |     - '    logger.go:42: 09:46:49 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.780192 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.780196 | controller |       found'
2026-03-14 09:56:26.780201 | controller |     - '    logger.go:42: 09:46:50 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.780205 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.780209 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.780214 | controller |     - '        '
2026-03-14 09:56:26.780218 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.780222 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.780226 | controller |     - '        '
2026-03-14 09:56:26.780231 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.780235 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.780239 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.780244 | controller |     - '          exit 1'
2026-03-14 09:56:26.780248 | controller |     - '        fi'
2026-03-14 09:56:26.780252 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.780257 | controller |     - '        '
2026-03-14 09:56:26.780263 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.780268 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.780272 | controller |       | base64 -d)'
2026-03-14 09:56:26.780276 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.780281 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.780285 | controller |     - '          exit 1'
2026-03-14 09:56:26.780289 | controller |     - '        fi'
2026-03-14 09:56:26.780293 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.780298 | controller |     - '        '
2026-03-14 09:56:26.780302 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.780306 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.780311 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.780316 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.780321 | controller |       --timeout=300s'
2026-03-14 09:56:26.780325 | controller |     - '        '
2026-03-14 09:56:26.780330 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.780334 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.780338 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.780343 | controller |     - '        '
2026-03-14 09:56:26.780347 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.780351 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.780356 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.780367 | controller |     - '        '
2026-03-14 09:56:26.780371 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.780375 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.780380 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.780384 | controller |       available"'
2026-03-14 09:56:26.780388 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.780393 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.780397 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.780401 | controller |     - '          exit 1'
2026-03-14 09:56:26.780405 | controller |     - '        fi'
2026-03-14 09:56:26.780410 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.780414 | controller |     - '        '
2026-03-14 09:56:26.780418 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.780424 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.780429 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.780433 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.780444 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.780448 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.780453 | controller |       -- \'
2026-03-14 09:56:26.780457 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.780461 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.780465 | controller |     - '    logger.go:42: 09:46:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.780470 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.780474 | controller |     - '    logger.go:42: 09:46:50 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.780478 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.780483 | controller |       found'
2026-03-14 09:56:26.780487 | controller |     - '    logger.go:42: 09:46:51 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.780491 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.780495 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.780500 | controller |     - '        '
2026-03-14 09:56:26.780504 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.780510 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.780514 | controller |     - '        '
2026-03-14 09:56:26.780518 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.780523 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.780527 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.780531 | controller |     - '          exit 1'
2026-03-14 09:56:26.780535 | controller |     - '        fi'
2026-03-14 09:56:26.780540 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.780544 | controller |     - '        '
2026-03-14 09:56:26.780548 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.780553 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.780557 | controller |       | base64 -d)'
2026-03-14 09:56:26.780561 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.780565 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.780570 | controller |     - '          exit 1'
2026-03-14 09:56:26.780574 | controller |     - '        fi'
2026-03-14 09:56:26.780578 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.780583 | controller |     - '        '
2026-03-14 09:56:26.780587 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.780591 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.780609 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.780613 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.780618 | controller |       --timeout=300s'
2026-03-14 09:56:26.780622 | controller |     - '        '
2026-03-14 09:56:26.780626 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.780633 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.780637 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.780641 | controller |     - '        '
2026-03-14 09:56:26.780646 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.780650 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.780654 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.780659 | controller |     - '        '
2026-03-14 09:56:26.780663 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.780667 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.780672 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.780676 | controller |       available"'
2026-03-14 09:56:26.780680 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.780685 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.780689 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.780693 | controller |     - '          exit 1'
2026-03-14 09:56:26.780697 | controller |     - '        fi'
2026-03-14 09:56:26.780702 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.780706 | controller |     - '        '
2026-03-14 09:56:26.780710 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.780715 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.780720 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.780725 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.780736 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.780741 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.780745 | controller |       -- \'
2026-03-14 09:56:26.780750 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.780754 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.780758 | controller |     - '    logger.go:42: 09:46:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.780763 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.780767 | controller |     - '    logger.go:42: 09:46:51 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.780771 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.780776 | controller |       found'
2026-03-14 09:56:26.780780 | controller |     - '    logger.go:42: 09:46:52 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.780784 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.780789 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.780795 | controller |     - '        '
2026-03-14 09:56:26.780802 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.780817 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.780823 | controller |     - '        '
2026-03-14 09:56:26.780829 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.780835 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.780841 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.780847 | controller |     - '          exit 1'
2026-03-14 09:56:26.780853 | controller |     - '        fi'
2026-03-14 09:56:26.780858 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.780864 | controller |     - '        '
2026-03-14 09:56:26.780869 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.780879 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.780885 | controller |       | base64 -d)'
2026-03-14 09:56:26.780891 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.780897 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.780902 | controller |     - '          exit 1'
2026-03-14 09:56:26.780908 | controller |     - '        fi'
2026-03-14 09:56:26.780912 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.780917 | controller |     - '        '
2026-03-14 09:56:26.780921 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.780925 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.780929 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.780933 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.780938 | controller |       --timeout=300s'
2026-03-14 09:56:26.780942 | controller |     - '        '
2026-03-14 09:56:26.780946 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.780950 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.780955 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.780962 | controller |     - '        '
2026-03-14 09:56:26.780966 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.780970 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.780974 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.780979 | controller |     - '        '
2026-03-14 09:56:26.780983 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.780987 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.780991 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.780995 | controller |       available"'
2026-03-14 09:56:26.781000 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.781004 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.781008 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.781012 | controller |     - '          exit 1'
2026-03-14 09:56:26.781016 | controller |     - '        fi'
2026-03-14 09:56:26.781021 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.781025 | controller |     - '        '
2026-03-14 09:56:26.781029 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.781033 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.781037 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.781042 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.781048 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.781062 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.781067 | controller |       -- \'
2026-03-14 09:56:26.781071 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.781076 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.781080 | controller |     - '    logger.go:42: 09:46:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.781084 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.781089 | controller |     - '    logger.go:42: 09:46:52 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.781093 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.781097 | controller |       found'
2026-03-14 09:56:26.781101 | controller |     - '    logger.go:42: 09:46:53 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.781105 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.781110 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.781114 | controller |     - '        '
2026-03-14 09:56:26.781118 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.781122 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.781126 | controller |     - '        '
2026-03-14 09:56:26.781131 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.781135 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.781139 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.781143 | controller |     - '          exit 1'
2026-03-14 09:56:26.781147 | controller |     - '        fi'
2026-03-14 09:56:26.781152 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.781158 | controller |     - '        '
2026-03-14 09:56:26.781162 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.781166 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.781170 | controller |       | base64 -d)'
2026-03-14 09:56:26.781174 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.781179 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.781183 | controller |     - '          exit 1'
2026-03-14 09:56:26.781187 | controller |     - '        fi'
2026-03-14 09:56:26.781191 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.781195 | controller |     - '        '
2026-03-14 09:56:26.781200 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.781204 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.781208 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.781212 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.781216 | controller |       --timeout=300s'
2026-03-14 09:56:26.781221 | controller |     - '        '
2026-03-14 09:56:26.781225 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.781229 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.781233 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.781238 | controller |     - '        '
2026-03-14 09:56:26.781242 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.781246 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.781250 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.781254 | controller |     - '        '
2026-03-14 09:56:26.781259 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.781263 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.781267 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.781271 | controller |       available"'
2026-03-14 09:56:26.781275 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.781280 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.781284 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.781288 | controller |     - '          exit 1'
2026-03-14 09:56:26.781292 | controller |     - '        fi'
2026-03-14 09:56:26.781296 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.781301 | controller |     - '        '
2026-03-14 09:56:26.781308 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.781313 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.781317 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.781321 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.781325 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.781336 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.781340 | controller |       -- \'
2026-03-14 09:56:26.781346 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.781351 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.781355 | controller |     - '    logger.go:42: 09:46:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.781359 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.781363 | controller |     - '    logger.go:42: 09:46:54 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.781368 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.781372 | controller |       found'
2026-03-14 09:56:26.781376 | controller |     - '    logger.go:42: 09:46:55 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.781380 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.781385 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.781389 | controller |     - '        '
2026-03-14 09:56:26.781393 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.781397 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.781402 | controller |     - '        '
2026-03-14 09:56:26.781406 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.781410 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.781414 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.781419 | controller |     - '          exit 1'
2026-03-14 09:56:26.781423 | controller |     - '        fi'
2026-03-14 09:56:26.781427 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.781431 | controller |     - '        '
2026-03-14 09:56:26.781436 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.781440 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.781444 | controller |       | base64 -d)'
2026-03-14 09:56:26.781448 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.781452 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.781457 | controller |     - '          exit 1'
2026-03-14 09:56:26.781461 | controller |     - '        fi'
2026-03-14 09:56:26.781465 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.781469 | controller |     - '        '
2026-03-14 09:56:26.781473 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.781478 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.781482 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.781486 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.781490 | controller |       --timeout=300s'
2026-03-14 09:56:26.781495 | controller |     - '        '
2026-03-14 09:56:26.781499 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.781503 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.781509 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.781514 | controller |     - '        '
2026-03-14 09:56:26.781518 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.781522 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.781526 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.781531 | controller |     - '        '
2026-03-14 09:56:26.781535 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.781541 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.781545 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.781549 | controller |       available"'
2026-03-14 09:56:26.781553 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.781558 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.781562 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.781566 | controller |     - '          exit 1'
2026-03-14 09:56:26.781570 | controller |     - '        fi'
2026-03-14 09:56:26.781575 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.781579 | controller |     - '        '
2026-03-14 09:56:26.781583 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.781587 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.781591 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.781632 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.781637 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.781641 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.781645 | controller |       -- \'
2026-03-14 09:56:26.781659 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.781664 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.781668 | controller |     - '    logger.go:42: 09:46:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.781673 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.781677 | controller |     - '    logger.go:42: 09:46:55 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.781681 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.781685 | controller |       found'
2026-03-14 09:56:26.781690 | controller |     - '    logger.go:42: 09:46:56 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.781694 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.781698 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.781702 | controller |     - '        '
2026-03-14 09:56:26.781707 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.781711 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.781715 | controller |     - '        '
2026-03-14 09:56:26.781719 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.781723 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.781728 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.781732 | controller |     - '          exit 1'
2026-03-14 09:56:26.781736 | controller |     - '        fi'
2026-03-14 09:56:26.781740 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.781744 | controller |     - '        '
2026-03-14 09:56:26.781749 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.781753 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.781757 | controller |       | base64 -d)'
2026-03-14 09:56:26.781761 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.781766 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.781774 | controller |     - '          exit 1'
2026-03-14 09:56:26.781778 | controller |     - '        fi'
2026-03-14 09:56:26.781782 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.781786 | controller |     - '        '
2026-03-14 09:56:26.781791 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.781795 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.781799 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.781803 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.781807 | controller |       --timeout=300s'
2026-03-14 09:56:26.781812 | controller |     - '        '
2026-03-14 09:56:26.781816 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.781820 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.781824 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.781829 | controller |     - '        '
2026-03-14 09:56:26.781833 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.781837 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.781841 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.781846 | controller |     - '        '
2026-03-14 09:56:26.781850 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.781854 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.781858 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.781862 | controller |       available"'
2026-03-14 09:56:26.781867 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.781871 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.781875 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.781879 | controller |     - '          exit 1'
2026-03-14 09:56:26.781884 | controller |     - '        fi'
2026-03-14 09:56:26.781888 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.781892 | controller |     - '        '
2026-03-14 09:56:26.781896 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.781901 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.781905 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.781909 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.781915 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.781919 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.781924 | controller |       -- \'
2026-03-14 09:56:26.781935 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.781940 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.781944 | controller |     - '    logger.go:42: 09:46:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.781949 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.781953 | controller |     - '    logger.go:42: 09:46:56 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.781957 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.781963 | controller |       found'
2026-03-14 09:56:26.781967 | controller |     - '    logger.go:42: 09:46:57 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.781972 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.781976 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.781980 | controller |     - '        '
2026-03-14 09:56:26.781984 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.781989 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.781993 | controller |     - '        '
2026-03-14 09:56:26.781997 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.782001 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.782005 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.782010 | controller |     - '          exit 1'
2026-03-14 09:56:26.782014 | controller |     - '        fi'
2026-03-14 09:56:26.782018 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.782023 | controller |     - '        '
2026-03-14 09:56:26.782027 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.782031 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.782037 | controller |       | base64 -d)'
2026-03-14 09:56:26.782041 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.782046 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.782050 | controller |     - '          exit 1'
2026-03-14 09:56:26.782054 | controller |     - '        fi'
2026-03-14 09:56:26.782058 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.782062 | controller |     - '        '
2026-03-14 09:56:26.782067 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.782071 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.782075 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.782079 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.782083 | controller |       --timeout=300s'
2026-03-14 09:56:26.782088 | controller |     - '        '
2026-03-14 09:56:26.782092 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.782096 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.782100 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.782105 | controller |     - '        '
2026-03-14 09:56:26.782109 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.782113 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.782117 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.782121 | controller |     - '        '
2026-03-14 09:56:26.782126 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.782130 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.782134 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.782138 | controller |       available"'
2026-03-14 09:56:26.782143 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.782147 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.782153 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.782171 | controller |     - '          exit 1'
2026-03-14 09:56:26.782175 | controller |     - '        fi'
2026-03-14 09:56:26.782180 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.782184 | controller |     - '        '
2026-03-14 09:56:26.782188 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.782192 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.782196 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.782201 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.782205 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.782209 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.782213 | controller |       -- \'
2026-03-14 09:56:26.782224 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.782228 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.782233 | controller |     - '    logger.go:42: 09:46:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.782237 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.782241 | controller |     - '    logger.go:42: 09:46:57 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.782248 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.782252 | controller |       found'
2026-03-14 09:56:26.782256 | controller |     - '    logger.go:42: 09:46:58 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.782260 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.782264 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.782269 | controller |     - '        '
2026-03-14 09:56:26.782273 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.782277 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.782281 | controller |     - '        '
2026-03-14 09:56:26.782285 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.782290 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.782294 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.782298 | controller |     - '          exit 1'
2026-03-14 09:56:26.782302 | controller |     - '        fi'
2026-03-14 09:56:26.782306 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.782311 | controller |     - '        '
2026-03-14 09:56:26.782315 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.782319 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.782323 | controller |       | base64 -d)'
2026-03-14 09:56:26.782328 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.782332 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.782336 | controller |     - '          exit 1'
2026-03-14 09:56:26.782341 | controller |     - '        fi'
2026-03-14 09:56:26.782347 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.782351 | controller |     - '        '
2026-03-14 09:56:26.782355 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.782361 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.782366 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.782370 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.782376 | controller |       --timeout=300s'
2026-03-14 09:56:26.782381 | controller |     - '        '
2026-03-14 09:56:26.782385 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.782389 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.782395 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.782399 | controller |     - '        '
2026-03-14 09:56:26.782404 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.782408 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.782412 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.782416 | controller |     - '        '
2026-03-14 09:56:26.782421 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.782425 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.782429 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.782433 | controller |       available"'
2026-03-14 09:56:26.782437 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.782442 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.782446 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.782450 | controller |     - '          exit 1'
2026-03-14 09:56:26.782454 | controller |     - '        fi'
2026-03-14 09:56:26.782458 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.782463 | controller |     - '        '
2026-03-14 09:56:26.782467 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.782471 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.782475 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.782482 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.782487 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.782491 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.782495 | controller |       -- \'
2026-03-14 09:56:26.782499 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.782510 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.782514 | controller |     - '    logger.go:42: 09:46:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.782518 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.782523 | controller |     - '    logger.go:42: 09:46:58 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.782527 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.782531 | controller |       found'
2026-03-14 09:56:26.782535 | controller |     - '    logger.go:42: 09:46:59 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.782539 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.782544 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.782548 | controller |     - '        '
2026-03-14 09:56:26.782552 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.782556 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.782563 | controller |     - '        '
2026-03-14 09:56:26.782567 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.782571 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.782576 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.782580 | controller |     - '          exit 1'
2026-03-14 09:56:26.782584 | controller |     - '        fi'
2026-03-14 09:56:26.782588 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.782592 | controller |     - '        '
2026-03-14 09:56:26.782610 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.782614 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.782618 | controller |       | base64 -d)'
2026-03-14 09:56:26.782623 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.782627 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.782631 | controller |     - '          exit 1'
2026-03-14 09:56:26.782635 | controller |     - '        fi'
2026-03-14 09:56:26.782639 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.782644 | controller |     - '        '
2026-03-14 09:56:26.782648 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.782652 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.782656 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.782660 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.782665 | controller |       --timeout=300s'
2026-03-14 09:56:26.782669 | controller |     - '        '
2026-03-14 09:56:26.782673 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.782677 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.782681 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.782686 | controller |     - '        '
2026-03-14 09:56:26.782690 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.782694 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.782698 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.782703 | controller |     - '        '
2026-03-14 09:56:26.782707 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.782711 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.782715 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.782719 | controller |       available"'
2026-03-14 09:56:26.782724 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.782728 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.782732 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.782736 | controller |     - '          exit 1'
2026-03-14 09:56:26.782740 | controller |     - '        fi'
2026-03-14 09:56:26.782745 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.782749 | controller |     - '        '
2026-03-14 09:56:26.782753 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.782757 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.782761 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.782767 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.782772 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.782776 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.782780 | controller |       -- \'
2026-03-14 09:56:26.782784 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.782789 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.782799 | controller |     - '    logger.go:42: 09:46:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.782804 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.782808 | controller |     - '    logger.go:42: 09:47:00 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.782813 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.782819 | controller |       found'
2026-03-14 09:56:26.782823 | controller |     - '    logger.go:42: 09:47:01 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.782827 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.782831 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.782836 | controller |     - '        '
2026-03-14 09:56:26.782840 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.782844 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.782848 | controller |     - '        '
2026-03-14 09:56:26.782852 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.782857 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.782861 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.782865 | controller |     - '          exit 1'
2026-03-14 09:56:26.782869 | controller |     - '        fi'
2026-03-14 09:56:26.782874 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.782878 | controller |     - '        '
2026-03-14 09:56:26.782882 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.782886 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.782890 | controller |       | base64 -d)'
2026-03-14 09:56:26.782895 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.782899 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.782903 | controller |     - '          exit 1'
2026-03-14 09:56:26.782907 | controller |     - '        fi'
2026-03-14 09:56:26.782911 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.782916 | controller |     - '        '
2026-03-14 09:56:26.782920 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.782924 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.782928 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.782932 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.782936 | controller |       --timeout=300s'
2026-03-14 09:56:26.782941 | controller |     - '        '
2026-03-14 09:56:26.782945 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.782949 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.782953 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.782959 | controller |     - '        '
2026-03-14 09:56:26.782963 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.782968 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.782972 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.782976 | controller |     - '        '
2026-03-14 09:56:26.782980 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.782985 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.782989 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.782993 | controller |       available"'
2026-03-14 09:56:26.782997 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.783001 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.783006 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.783010 | controller |     - '          exit 1'
2026-03-14 09:56:26.783014 | controller |     - '        fi'
2026-03-14 09:56:26.783018 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.783022 | controller |     - '        '
2026-03-14 09:56:26.783040 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.783046 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.783051 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.783055 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.783059 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.783064 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.783068 | controller |       -- \'
2026-03-14 09:56:26.783072 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.783077 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.783088 | controller |     - '    logger.go:42: 09:47:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.783092 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.783097 | controller |     - '    logger.go:42: 09:47:01 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.783101 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.783105 | controller |       found'
2026-03-14 09:56:26.783109 | controller |     - '    logger.go:42: 09:47:02 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.783114 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.783118 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.783122 | controller |     - '        '
2026-03-14 09:56:26.783127 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.783131 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.783135 | controller |     - '        '
2026-03-14 09:56:26.783139 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.783144 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.783151 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.783155 | controller |     - '          exit 1'
2026-03-14 09:56:26.783166 | controller |     - '        fi'
2026-03-14 09:56:26.783171 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.783179 | controller |     - '        '
2026-03-14 09:56:26.783185 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.783189 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.783194 | controller |       | base64 -d)'
2026-03-14 09:56:26.783198 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.783202 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.783207 | controller |     - '          exit 1'
2026-03-14 09:56:26.783211 | controller |     - '        fi'
2026-03-14 09:56:26.783215 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.783220 | controller |     - '        '
2026-03-14 09:56:26.783224 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.783228 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.783233 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.783237 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.783241 | controller |       --timeout=300s'
2026-03-14 09:56:26.783246 | controller |     - '        '
2026-03-14 09:56:26.783250 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.783254 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.783258 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.783263 | controller |     - '        '
2026-03-14 09:56:26.783267 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.783271 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.783276 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.783280 | controller |     - '        '
2026-03-14 09:56:26.783284 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.783289 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.783293 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.783297 | controller |       available"'
2026-03-14 09:56:26.783302 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.783306 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.783310 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.783315 | controller |     - '          exit 1'
2026-03-14 09:56:26.783319 | controller |     - '        fi'
2026-03-14 09:56:26.783323 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.783328 | controller |     - '        '
2026-03-14 09:56:26.783332 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.783336 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.783341 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.783345 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.783349 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.783354 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.783360 | controller |       -- \'
2026-03-14 09:56:26.783364 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.783370 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.783375 | controller |     - '    logger.go:42: 09:47:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.783386 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.783391 | controller |     - '    logger.go:42: 09:47:02 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.783395 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.783400 | controller |       found'
2026-03-14 09:56:26.783406 | controller |     - '    logger.go:42: 09:47:03 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.783412 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.783417 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.783421 | controller |     - '        '
2026-03-14 09:56:26.783425 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.783430 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.783434 | controller |     - '        '
2026-03-14 09:56:26.783438 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.783442 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.783447 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.783451 | controller |     - '          exit 1'
2026-03-14 09:56:26.783455 | controller |     - '        fi'
2026-03-14 09:56:26.783460 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.783464 | controller |     - '        '
2026-03-14 09:56:26.783468 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.783474 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.783479 | controller |       | base64 -d)'
2026-03-14 09:56:26.783483 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.783488 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.783492 | controller |     - '          exit 1'
2026-03-14 09:56:26.783498 | controller |     - '        fi'
2026-03-14 09:56:26.783502 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.783506 | controller |     - '        '
2026-03-14 09:56:26.783511 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.783515 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.783519 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.783524 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.783528 | controller |       --timeout=300s'
2026-03-14 09:56:26.783532 | controller |     - '        '
2026-03-14 09:56:26.783537 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.783541 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.783545 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.783551 | controller |     - '        '
2026-03-14 09:56:26.783556 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.783560 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.783564 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.783569 | controller |     - '        '
2026-03-14 09:56:26.783573 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.783579 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.783584 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.783588 | controller |       available"'
2026-03-14 09:56:26.783592 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.783609 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.783614 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.783618 | controller |     - '          exit 1'
2026-03-14 09:56:26.783623 | controller |     - '        fi'
2026-03-14 09:56:26.783627 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.783631 | controller |     - '        '
2026-03-14 09:56:26.783636 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.783640 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.783644 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.783648 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.783653 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.783657 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.783661 | controller |       -- \'
2026-03-14 09:56:26.783666 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.783670 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.783675 | controller |     - '    logger.go:42: 09:47:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.783679 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.783698 | controller |     - '    logger.go:42: 09:47:03 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.783703 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.783718 | controller |       found'
2026-03-14 09:56:26.783723 | controller |     - '    logger.go:42: 09:47:04 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.783727 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.783736 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.783744 | controller |     - '        '
2026-03-14 09:56:26.783752 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.783758 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.783763 | controller |     - '        '
2026-03-14 09:56:26.783768 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.783774 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.783779 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.783786 | controller |     - '          exit 1'
2026-03-14 09:56:26.783793 | controller |     - '        fi'
2026-03-14 09:56:26.783797 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.783803 | controller |     - '        '
2026-03-14 09:56:26.783808 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.783816 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.783831 | controller |       | base64 -d)'
2026-03-14 09:56:26.783839 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.783850 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.783855 | controller |     - '          exit 1'
2026-03-14 09:56:26.783859 | controller |     - '        fi'
2026-03-14 09:56:26.783875 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.783890 | controller |     - '        '
2026-03-14 09:56:26.783894 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.783899 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.783904 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.783914 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.783919 | controller |       --timeout=300s'
2026-03-14 09:56:26.783928 | controller |     - '        '
2026-03-14 09:56:26.783933 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.783937 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.783942 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.783946 | controller |     - '        '
2026-03-14 09:56:26.783950 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.783955 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.783959 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.783963 | controller |     - '        '
2026-03-14 09:56:26.783967 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.783972 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.783976 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.783980 | controller |       available"'
2026-03-14 09:56:26.783984 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.783988 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.783993 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.783997 | controller |     - '          exit 1'
2026-03-14 09:56:26.784001 | controller |     - '        fi'
2026-03-14 09:56:26.784005 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.784009 | controller |     - '        '
2026-03-14 09:56:26.784014 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.784018 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.784022 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.784026 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.784030 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.784035 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.784039 | controller |       -- \'
2026-03-14 09:56:26.784043 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.784047 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.784051 | controller |     - '    logger.go:42: 09:47:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.784056 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.784066 | controller |     - '    logger.go:42: 09:47:04 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.784070 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.784078 | controller |       found'
2026-03-14 09:56:26.784082 | controller |     - '    logger.go:42: 09:47:05 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.784086 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.784091 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.784095 | controller |     - '        '
2026-03-14 09:56:26.784099 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.784106 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.784110 | controller |     - '        '
2026-03-14 09:56:26.784114 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.784118 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.784122 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.784132 | controller |     - '          exit 1'
2026-03-14 09:56:26.784136 | controller |     - '        fi'
2026-03-14 09:56:26.784141 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.784145 | controller |     - '        '
2026-03-14 09:56:26.784149 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.784153 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.784158 | controller |       | base64 -d)'
2026-03-14 09:56:26.784162 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.784166 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.784170 | controller |     - '          exit 1'
2026-03-14 09:56:26.784175 | controller |     - '        fi'
2026-03-14 09:56:26.784184 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.784188 | controller |     - '        '
2026-03-14 09:56:26.784192 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.784196 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.784201 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.784205 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.784209 | controller |       --timeout=300s'
2026-03-14 09:56:26.784213 | controller |     - '        '
2026-03-14 09:56:26.784218 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.784222 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.784226 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.784230 | controller |     - '        '
2026-03-14 09:56:26.784235 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.784239 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.784243 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.784247 | controller |     - '        '
2026-03-14 09:56:26.784252 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.784256 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.784260 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.784264 | controller |       available"'
2026-03-14 09:56:26.784269 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.784273 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.784277 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.784283 | controller |     - '          exit 1'
2026-03-14 09:56:26.784287 | controller |     - '        fi'
2026-03-14 09:56:26.784291 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.784296 | controller |     - '        '
2026-03-14 09:56:26.784300 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.784304 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.784308 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.784313 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.784317 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.784321 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.784325 | controller |       -- \'
2026-03-14 09:56:26.784330 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.784334 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.784338 | controller |     - '    logger.go:42: 09:47:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.784342 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.784347 | controller |     - '    logger.go:42: 09:47:05 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.784357 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.784362 | controller |       found'
2026-03-14 09:56:26.784366 | controller |     - '    logger.go:42: 09:47:06 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.784370 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.784375 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.784379 | controller |     - '        '
2026-03-14 09:56:26.784383 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.784387 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.784393 | controller |     - '        '
2026-03-14 09:56:26.784398 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.784402 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.784406 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.784410 | controller |     - '          exit 1'
2026-03-14 09:56:26.784415 | controller |     - '        fi'
2026-03-14 09:56:26.784419 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.784423 | controller |     - '        '
2026-03-14 09:56:26.784427 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.784432 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.784436 | controller |       | base64 -d)'
2026-03-14 09:56:26.784440 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.784444 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.784448 | controller |     - '          exit 1'
2026-03-14 09:56:26.784453 | controller |     - '        fi'
2026-03-14 09:56:26.784457 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.784461 | controller |     - '        '
2026-03-14 09:56:26.784465 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.784469 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.784474 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.784478 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.784484 | controller |       --timeout=300s'
2026-03-14 09:56:26.784488 | controller |     - '        '
2026-03-14 09:56:26.784492 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.784497 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.784501 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.784505 | controller |     - '        '
2026-03-14 09:56:26.784509 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.784513 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.784518 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.784522 | controller |     - '        '
2026-03-14 09:56:26.784526 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.784530 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.784534 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.784539 | controller |       available"'
2026-03-14 09:56:26.784543 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.784547 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.784551 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.784555 | controller |     - '          exit 1'
2026-03-14 09:56:26.784560 | controller |     - '        fi'
2026-03-14 09:56:26.784564 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.784568 | controller |     - '        '
2026-03-14 09:56:26.784572 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.784577 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.784581 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.784585 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.784589 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.784609 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.784613 | controller |       -- \'
2026-03-14 09:56:26.784618 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.784622 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.784626 | controller |     - '    logger.go:42: 09:47:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.784630 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.784635 | controller |     - '    logger.go:42: 09:47:07 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.784639 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.784650 | controller |       found'
2026-03-14 09:56:26.784654 | controller |     - '    logger.go:42: 09:47:08 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.784658 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.784663 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.784667 | controller |     - '        '
2026-03-14 09:56:26.784671 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.784676 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.784684 | controller |     - '        '
2026-03-14 09:56:26.784690 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.784694 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.784699 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.784703 | controller |     - '          exit 1'
2026-03-14 09:56:26.784707 | controller |     - '        fi'
2026-03-14 09:56:26.784711 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.784715 | controller |     - '        '
2026-03-14 09:56:26.784725 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.784729 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.784739 | controller |       | base64 -d)'
2026-03-14 09:56:26.784743 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.784747 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.784752 | controller |     - '          exit 1'
2026-03-14 09:56:26.784756 | controller |     - '        fi'
2026-03-14 09:56:26.784760 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.784764 | controller |     - '        '
2026-03-14 09:56:26.784769 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.784773 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.784777 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.784781 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.784786 | controller |       --timeout=300s'
2026-03-14 09:56:26.784790 | controller |     - '        '
2026-03-14 09:56:26.784794 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.784800 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.784804 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.784809 | controller |     - '        '
2026-03-14 09:56:26.784813 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.784817 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.784821 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.784826 | controller |     - '        '
2026-03-14 09:56:26.784830 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.784834 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.784838 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.784843 | controller |       available"'
2026-03-14 09:56:26.784847 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.784851 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.784857 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.784861 | controller |     - '          exit 1'
2026-03-14 09:56:26.784866 | controller |     - '        fi'
2026-03-14 09:56:26.784870 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.784874 | controller |     - '        '
2026-03-14 09:56:26.784878 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.784883 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.784887 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.784893 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.784897 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.784901 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.784906 | controller |       -- \'
2026-03-14 09:56:26.784910 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.784914 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.784918 | controller |     - '    logger.go:42: 09:47:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.784923 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.784927 | controller |     - '    logger.go:42: 09:47:08 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.784931 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.784935 | controller |       found'
2026-03-14 09:56:26.784946 | controller |     - '    logger.go:42: 09:47:09 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.784951 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.784955 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.784960 | controller |     - '        '
2026-03-14 09:56:26.784964 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.784968 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.784972 | controller |     - '        '
2026-03-14 09:56:26.784977 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.784981 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.784985 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.784989 | controller |     - '          exit 1'
2026-03-14 09:56:26.784994 | controller |     - '        fi'
2026-03-14 09:56:26.784998 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.785002 | controller |     - '        '
2026-03-14 09:56:26.785006 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.785010 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.785015 | controller |       | base64 -d)'
2026-03-14 09:56:26.785019 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.785023 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.785027 | controller |     - '          exit 1'
2026-03-14 09:56:26.785032 | controller |     - '        fi'
2026-03-14 09:56:26.785036 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.785040 | controller |     - '        '
2026-03-14 09:56:26.785044 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.785048 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.785053 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.785057 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.785061 | controller |       --timeout=300s'
2026-03-14 09:56:26.785065 | controller |     - '        '
2026-03-14 09:56:26.785070 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.785074 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.785078 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.785082 | controller |     - '        '
2026-03-14 09:56:26.785087 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.785093 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.785097 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.785101 | controller |     - '        '
2026-03-14 09:56:26.785105 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.785110 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.785114 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.785118 | controller |       available"'
2026-03-14 09:56:26.785122 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.785126 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.785131 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.785135 | controller |     - '          exit 1'
2026-03-14 09:56:26.785139 | controller |     - '        fi'
2026-03-14 09:56:26.785143 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.785148 | controller |     - '        '
2026-03-14 09:56:26.785152 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.785156 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.785160 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.785164 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.785169 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.785173 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.785177 | controller |       -- \'
2026-03-14 09:56:26.785181 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.785186 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.785190 | controller |     - '    logger.go:42: 09:47:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.785194 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.785198 | controller |     - '    logger.go:42: 09:47:09 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.785203 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.785207 | controller |       found'
2026-03-14 09:56:26.785211 | controller |     - '    logger.go:42: 09:47:10 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.785221 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.785231 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.785235 | controller |     - '        '
2026-03-14 09:56:26.785239 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.785244 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.785248 | controller |     - '        '
2026-03-14 09:56:26.785252 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.785258 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.785263 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.785267 | controller |     - '          exit 1'
2026-03-14 09:56:26.785271 | controller |     - '        fi'
2026-03-14 09:56:26.785275 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.785279 | controller |     - '        '
2026-03-14 09:56:26.785284 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.785289 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.785294 | controller |       | base64 -d)'
2026-03-14 09:56:26.785298 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.785302 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.785306 | controller |     - '          exit 1'
2026-03-14 09:56:26.785311 | controller |     - '        fi'
2026-03-14 09:56:26.785315 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.785319 | controller |     - '        '
2026-03-14 09:56:26.785323 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.785327 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.785332 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.785336 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.785340 | controller |       --timeout=300s'
2026-03-14 09:56:26.785344 | controller |     - '        '
2026-03-14 09:56:26.785348 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.785352 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.785357 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.785362 | controller |     - '        '
2026-03-14 09:56:26.785367 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.785372 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.785376 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.785381 | controller |     - '        '
2026-03-14 09:56:26.785386 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.785391 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.785395 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.785400 | controller |       available"'
2026-03-14 09:56:26.785404 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.785408 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.785412 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.785416 | controller |     - '          exit 1'
2026-03-14 09:56:26.785421 | controller |     - '        fi'
2026-03-14 09:56:26.785425 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.785429 | controller |     - '        '
2026-03-14 09:56:26.785433 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.785437 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.785442 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.785446 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.785450 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.785454 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.785458 | controller |       -- \'
2026-03-14 09:56:26.785463 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.785470 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.785474 | controller |     - '    logger.go:42: 09:47:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.785479 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.785483 | controller |     - '    logger.go:42: 09:47:10 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.785487 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.785491 | controller |       found'
2026-03-14 09:56:26.785495 | controller |     - '    logger.go:42: 09:47:11 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.785500 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.785504 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.785515 | controller |     - '        '
2026-03-14 09:56:26.785519 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.785523 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.785528 | controller |     - '        '
2026-03-14 09:56:26.785532 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.785536 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.785540 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.785545 | controller |     - '          exit 1'
2026-03-14 09:56:26.785549 | controller |     - '        fi'
2026-03-14 09:56:26.785553 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.785557 | controller |     - '        '
2026-03-14 09:56:26.785561 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.785566 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.785570 | controller |       | base64 -d)'
2026-03-14 09:56:26.785574 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.785578 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.785582 | controller |     - '          exit 1'
2026-03-14 09:56:26.785587 | controller |     - '        fi'
2026-03-14 09:56:26.785591 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.785608 | controller |     - '        '
2026-03-14 09:56:26.785612 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.785617 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.785621 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.785625 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.785629 | controller |       --timeout=300s'
2026-03-14 09:56:26.785634 | controller |     - '        '
2026-03-14 09:56:26.785638 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.785642 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.785646 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.785651 | controller |     - '        '
2026-03-14 09:56:26.785655 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.785659 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.785666 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.785671 | controller |     - '        '
2026-03-14 09:56:26.785675 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.785679 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.785693 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.785697 | controller |       available"'
2026-03-14 09:56:26.785703 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.785707 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.785711 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.785716 | controller |     - '          exit 1'
2026-03-14 09:56:26.785720 | controller |     - '        fi'
2026-03-14 09:56:26.785724 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.785728 | controller |     - '        '
2026-03-14 09:56:26.785733 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.785737 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.785741 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.785745 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.785749 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.785754 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.785758 | controller |       -- \'
2026-03-14 09:56:26.785762 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.785766 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.785770 | controller |     - '    logger.go:42: 09:47:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.785775 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.785779 | controller |     - '    logger.go:42: 09:47:11 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.785783 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.785787 | controller |       found'
2026-03-14 09:56:26.785791 | controller |     - '    logger.go:42: 09:47:12 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.785796 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.785800 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.785804 | controller |     - '        '
2026-03-14 09:56:26.785815 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.785820 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.785824 | controller |     - '        '
2026-03-14 09:56:26.785828 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.785833 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.785837 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.785841 | controller |     - '          exit 1'
2026-03-14 09:56:26.785845 | controller |     - '        fi'
2026-03-14 09:56:26.785850 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.785854 | controller |     - '        '
2026-03-14 09:56:26.785858 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.785862 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.785867 | controller |       | base64 -d)'
2026-03-14 09:56:26.785871 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.785875 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.785879 | controller |     - '          exit 1'
2026-03-14 09:56:26.785884 | controller |     - '        fi'
2026-03-14 09:56:26.785888 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.785894 | controller |     - '        '
2026-03-14 09:56:26.785898 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.785902 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.785908 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.785913 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.785917 | controller |       --timeout=300s'
2026-03-14 09:56:26.785921 | controller |     - '        '
2026-03-14 09:56:26.785925 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.785930 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.785934 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.785938 | controller |     - '        '
2026-03-14 09:56:26.785943 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.785947 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.785951 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.785955 | controller |     - '        '
2026-03-14 09:56:26.785960 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.785964 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.785968 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.785972 | controller |       available"'
2026-03-14 09:56:26.785976 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.785981 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.785985 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.785989 | controller |     - '          exit 1'
2026-03-14 09:56:26.785993 | controller |     - '        fi'
2026-03-14 09:56:26.785998 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.786002 | controller |     - '        '
2026-03-14 09:56:26.786006 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.786010 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.786015 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.786019 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.786023 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.786027 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.786032 | controller |       -- \'
2026-03-14 09:56:26.786036 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.786040 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.786044 | controller |     - '    logger.go:42: 09:47:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.786049 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.786053 | controller |     - '    logger.go:42: 09:47:13 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.786057 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.786061 | controller |       found'
2026-03-14 09:56:26.786067 | controller |     - '    logger.go:42: 09:47:14 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.786072 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.786076 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.786080 | controller |     - '        '
2026-03-14 09:56:26.786085 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.786095 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.786099 | controller |     - '        '
2026-03-14 09:56:26.786104 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.786108 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.786112 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.786117 | controller |     - '          exit 1'
2026-03-14 09:56:26.786121 | controller |     - '        fi'
2026-03-14 09:56:26.786125 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.786131 | controller |     - '        '
2026-03-14 09:56:26.786135 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.786141 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.786147 | controller |       | base64 -d)'
2026-03-14 09:56:26.786151 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.786155 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.786160 | controller |     - '          exit 1'
2026-03-14 09:56:26.786164 | controller |     - '        fi'
2026-03-14 09:56:26.786168 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.786172 | controller |     - '        '
2026-03-14 09:56:26.786177 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.786181 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.786185 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.786189 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.786194 | controller |       --timeout=300s'
2026-03-14 09:56:26.786198 | controller |     - '        '
2026-03-14 09:56:26.786202 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.786206 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.786210 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.786215 | controller |     - '        '
2026-03-14 09:56:26.786219 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.786223 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.786227 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.786232 | controller |     - '        '
2026-03-14 09:56:26.786236 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.786240 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.786244 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.786249 | controller |       available"'
2026-03-14 09:56:26.786253 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.786257 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.786261 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.786266 | controller |     - '          exit 1'
2026-03-14 09:56:26.786270 | controller |     - '        fi'
2026-03-14 09:56:26.786276 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.786280 | controller |     - '        '
2026-03-14 09:56:26.786284 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.786288 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.786293 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.786297 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.786301 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.786305 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.786310 | controller |       -- \'
2026-03-14 09:56:26.786314 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.786318 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.786322 | controller |     - '    logger.go:42: 09:47:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.786326 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.786331 | controller |     - '    logger.go:42: 09:47:14 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.786335 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.786339 | controller |       found'
2026-03-14 09:56:26.786344 | controller |     - '    logger.go:42: 09:47:15 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.786348 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.786352 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.786356 | controller |     - '        '
2026-03-14 09:56:26.786361 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.786371 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.786375 | controller |     - '        '
2026-03-14 09:56:26.786379 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.786384 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.786388 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.786392 | controller |     - '          exit 1'
2026-03-14 09:56:26.786396 | controller |     - '        fi'
2026-03-14 09:56:26.786401 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.786405 | controller |     - '        '
2026-03-14 09:56:26.786409 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.786413 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.786418 | controller |       | base64 -d)'
2026-03-14 09:56:26.786422 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.786426 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.786430 | controller |     - '          exit 1'
2026-03-14 09:56:26.786435 | controller |     - '        fi'
2026-03-14 09:56:26.786439 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.786443 | controller |     - '        '
2026-03-14 09:56:26.786447 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.786452 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.786456 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.786462 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.786466 | controller |       --timeout=300s'
2026-03-14 09:56:26.786470 | controller |     - '        '
2026-03-14 09:56:26.786476 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.786480 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.786485 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.786489 | controller |     - '        '
2026-03-14 09:56:26.786493 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.786497 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.786502 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.786506 | controller |     - '        '
2026-03-14 09:56:26.786510 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.786514 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.786519 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.786523 | controller |       available"'
2026-03-14 09:56:26.786527 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.786531 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.786536 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.786540 | controller |     - '          exit 1'
2026-03-14 09:56:26.786544 | controller |     - '        fi'
2026-03-14 09:56:26.786548 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.786552 | controller |     - '        '
2026-03-14 09:56:26.786557 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.786561 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.786565 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.786569 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.786574 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.786578 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.786582 | controller |       -- \'
2026-03-14 09:56:26.786586 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.786592 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.786609 | controller |     - '    logger.go:42: 09:47:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.786614 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.786618 | controller |     - '    logger.go:42: 09:47:15 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.786622 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.786627 | controller |       found'
2026-03-14 09:56:26.786631 | controller |     - '    logger.go:42: 09:47:16 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.786635 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.786640 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.786644 | controller |     - '        '
2026-03-14 09:56:26.786648 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.786653 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.786657 | controller |     - '        '
2026-03-14 09:56:26.786667 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.786674 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.786678 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.786682 | controller |     - '          exit 1'
2026-03-14 09:56:26.786687 | controller |     - '        fi'
2026-03-14 09:56:26.786691 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.786695 | controller |     - '        '
2026-03-14 09:56:26.786700 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.786704 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.786708 | controller |       | base64 -d)'
2026-03-14 09:56:26.786713 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.786717 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.786721 | controller |     - '          exit 1'
2026-03-14 09:56:26.786725 | controller |     - '        fi'
2026-03-14 09:56:26.786730 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.786737 | controller |     - '        '
2026-03-14 09:56:26.786743 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.786747 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.786751 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.786756 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.786760 | controller |       --timeout=300s'
2026-03-14 09:56:26.786764 | controller |     - '        '
2026-03-14 09:56:26.786768 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.786773 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.786777 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.786781 | controller |     - '        '
2026-03-14 09:56:26.786786 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.786790 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.786794 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.786798 | controller |     - '        '
2026-03-14 09:56:26.786803 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.786807 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.786811 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.786816 | controller |       available"'
2026-03-14 09:56:26.786820 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.786824 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.786828 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.786832 | controller |     - '          exit 1'
2026-03-14 09:56:26.786837 | controller |     - '        fi'
2026-03-14 09:56:26.786841 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.786845 | controller |     - '        '
2026-03-14 09:56:26.786850 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.786854 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.786858 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.786862 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.786868 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.786873 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.786877 | controller |       -- \'
2026-03-14 09:56:26.786881 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.786885 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.786890 | controller |     - '    logger.go:42: 09:47:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.786894 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.786898 | controller |     - '    logger.go:42: 09:47:16 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.786903 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.786907 | controller |       found'
2026-03-14 09:56:26.786911 | controller |     - '    logger.go:42: 09:47:17 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.786916 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.786922 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.786926 | controller |     - '        '
2026-03-14 09:56:26.786930 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.786935 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.786939 | controller |     - '        '
2026-03-14 09:56:26.786950 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.786954 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.786959 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.786963 | controller |     - '          exit 1'
2026-03-14 09:56:26.786968 | controller |     - '        fi'
2026-03-14 09:56:26.786972 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.786976 | controller |     - '        '
2026-03-14 09:56:26.786981 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.786985 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.786989 | controller |       | base64 -d)'
2026-03-14 09:56:26.786993 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.786998 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.787002 | controller |     - '          exit 1'
2026-03-14 09:56:26.787006 | controller |     - '        fi'
2026-03-14 09:56:26.787010 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.787015 | controller |     - '        '
2026-03-14 09:56:26.787019 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.787023 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.787049 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.787054 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.787061 | controller |       --timeout=300s'
2026-03-14 09:56:26.787065 | controller |     - '        '
2026-03-14 09:56:26.787069 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.787073 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.787078 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.787082 | controller |     - '        '
2026-03-14 09:56:26.787086 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.787093 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.787097 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.787101 | controller |     - '        '
2026-03-14 09:56:26.787106 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.787110 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.787114 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.787119 | controller |       available"'
2026-03-14 09:56:26.787123 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.787127 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.787131 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.787136 | controller |     - '          exit 1'
2026-03-14 09:56:26.787140 | controller |     - '        fi'
2026-03-14 09:56:26.787144 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.787148 | controller |     - '        '
2026-03-14 09:56:26.787153 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.787157 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.787161 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.787166 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.787170 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.787174 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.787179 | controller |       -- \'
2026-03-14 09:56:26.787183 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.787187 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.787191 | controller |     - '    logger.go:42: 09:47:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.787196 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.787200 | controller |     - '    logger.go:42: 09:47:17 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.787204 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.787209 | controller |       found'
2026-03-14 09:56:26.787213 | controller |     - '    logger.go:42: 09:47:18 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.787217 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.787221 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.787226 | controller |     - '        '
2026-03-14 09:56:26.787230 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.787234 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.787239 | controller |     - '        '
2026-03-14 09:56:26.787243 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.787254 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.787258 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.787263 | controller |     - '          exit 1'
2026-03-14 09:56:26.787267 | controller |     - '        fi'
2026-03-14 09:56:26.787271 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.787276 | controller |     - '        '
2026-03-14 09:56:26.787280 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.787284 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.787290 | controller |       | base64 -d)'
2026-03-14 09:56:26.787295 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.787299 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.787303 | controller |     - '          exit 1'
2026-03-14 09:56:26.787307 | controller |     - '        fi'
2026-03-14 09:56:26.787312 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.787316 | controller |     - '        '
2026-03-14 09:56:26.787320 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.787325 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.787329 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.787333 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.787337 | controller |       --timeout=300s'
2026-03-14 09:56:26.787342 | controller |     - '        '
2026-03-14 09:56:26.787346 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.787350 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.787355 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.787359 | controller |     - '        '
2026-03-14 09:56:26.787363 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.787367 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.787372 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.787376 | controller |     - '        '
2026-03-14 09:56:26.787380 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.787385 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.787389 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.787393 | controller |       available"'
2026-03-14 09:56:26.787398 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.787402 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.787408 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.787412 | controller |     - '          exit 1'
2026-03-14 09:56:26.787417 | controller |     - '        fi'
2026-03-14 09:56:26.787421 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.787425 | controller |     - '        '
2026-03-14 09:56:26.787429 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.787434 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.787438 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.787442 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.787447 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.787451 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.787457 | controller |       -- \'
2026-03-14 09:56:26.787461 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.787466 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.787471 | controller |     - '    logger.go:42: 09:47:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.787476 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.787480 | controller |     - '    logger.go:42: 09:47:18 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.787484 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.787489 | controller |       found'
2026-03-14 09:56:26.787493 | controller |     - '    logger.go:42: 09:47:19 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.787497 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.787501 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.787506 | controller |     - '        '
2026-03-14 09:56:26.787510 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.787514 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.787518 | controller |     - '        '
2026-03-14 09:56:26.787523 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.787527 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.787537 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.787541 | controller |     - '          exit 1'
2026-03-14 09:56:26.787546 | controller |     - '        fi'
2026-03-14 09:56:26.787550 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.787554 | controller |     - '        '
2026-03-14 09:56:26.787559 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.787563 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.787567 | controller |       | base64 -d)'
2026-03-14 09:56:26.787571 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.787576 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.787580 | controller |     - '          exit 1'
2026-03-14 09:56:26.787584 | controller |     - '        fi'
2026-03-14 09:56:26.787588 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.787593 | controller |     - '        '
2026-03-14 09:56:26.787610 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.787614 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.787619 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.787623 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.787627 | controller |       --timeout=300s'
2026-03-14 09:56:26.787632 | controller |     - '        '
2026-03-14 09:56:26.787636 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.787640 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.787644 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.787649 | controller |     - '        '
2026-03-14 09:56:26.787653 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.787657 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.787661 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.787668 | controller |     - '        '
2026-03-14 09:56:26.787672 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.787678 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.787682 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.787688 | controller |       available"'
2026-03-14 09:56:26.787692 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.787697 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.787701 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.787705 | controller |     - '          exit 1'
2026-03-14 09:56:26.787709 | controller |     - '        fi'
2026-03-14 09:56:26.787713 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.787718 | controller |     - '        '
2026-03-14 09:56:26.787722 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.787726 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.787730 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.787734 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.787739 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.787743 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.787747 | controller |       -- \'
2026-03-14 09:56:26.787751 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.787756 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.787760 | controller |     - '    logger.go:42: 09:47:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.787764 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.787768 | controller |     - '    logger.go:42: 09:47:20 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.787773 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.787777 | controller |       found'
2026-03-14 09:56:26.787781 | controller |     - '    logger.go:42: 09:47:21 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:56:26.787785 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:56:26.787789 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:56:26.787794 | controller |     - '        '
2026-03-14 09:56:26.787798 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:56:26.787802 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:56:26.787806 | controller |     - '        '
2026-03-14 09:56:26.787811 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:56:26.787815 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:56:26.787819 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:56:26.787830 | controller |     - '          exit 1'
2026-03-14 09:56:26.787837 | controller |     - '        fi'
2026-03-14 09:56:26.787841 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:56:26.787846 | controller |     - '        '
2026-03-14 09:56:26.787850 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:56:26.787854 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:56:26.787858 | controller |       | base64 -d)'
2026-03-14 09:56:26.787863 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:56:26.787867 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:56:26.787871 | controller |     - '          exit 1'
2026-03-14 09:56:26.787875 | controller |     - '        fi'
2026-03-14 09:56:26.787880 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:56:26.787884 | controller |     - '        '
2026-03-14 09:56:26.787890 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:56:26.787894 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:56:26.787898 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:56:26.787903 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:56:26.787907 | controller |       --timeout=300s'
2026-03-14 09:56:26.787911 | controller |     - '        '
2026-03-14 09:56:26.787915 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:56:26.787919 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:56:26.787924 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:56:26.787928 | controller |     - '        '
2026-03-14 09:56:26.787932 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.787936 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.787941 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:56:26.787945 | controller |     - '        '
2026-03-14 09:56:26.787949 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:56:26.787953 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:56:26.787958 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:56:26.787962 | controller |       available"'
2026-03-14 09:56:26.787966 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:56:26.787970 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:56:26.787974 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:56:26.787978 | controller |     - '          exit 1'
2026-03-14 09:56:26.787983 | controller |     - '        fi'
2026-03-14 09:56:26.787987 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:56:26.787991 | controller |     - '        '
2026-03-14 09:56:26.787996 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:56:26.788000 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:56:26.788004 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.788008 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:56:26.788012 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.788017 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:56:26.788021 | controller |       -- \'
2026-03-14 09:56:26.788025 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:56:26.788029 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:56:26.788034 | controller |     - '    logger.go:42: 09:47:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:56:26.788038 | controller |       ac-watcher to be Ready...'
2026-03-14 09:56:26.788042 | controller |     - '    logger.go:42: 09:47:21 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:56:26.788046 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:56:26.788050 | controller |       found'
2026-03-14 09:56:26.788055 | controller |     - '    logger.go:42: 09:47:22 | appcred-tests/2-deploy-appcred | test step failed
2026-03-14 09:56:26.788062 | controller |       2-deploy-appcred'
2026-03-14 09:56:26.788066 | controller |     - '    case.go:396: failed in step 2-deploy-appcred'
2026-03-14 09:56:26.788070 | controller |     - '    case.go:398: command "NS=\"${NAMESPACE}\"\\n echo \"Waiting for KeystoneApplicationCredential..."
2026-03-14 09:56:26.788075 | controller |       failed, exit status 1'
2026-03-14 09:56:26.788079 | controller |     - '    logger.go:42: 09:47:22 | appcred-tests | skipping kubernetes event logging'
2026-03-14 09:56:26.788083 | controller |     - === CONT  kuttl/harness/watcher-notification
2026-03-14 09:56:26.788087 | controller |     - '    logger.go:42: 09:47:22 | watcher-notification | Skipping creation of user-supplied
2026-03-14 09:56:26.788091 | controller |       namespace: watcher-kuttl-default'
2026-03-14 09:56:26.788102 | controller |     - '    logger.go:42: 09:47:22 | watcher-notification/0-cleanup-watcher | starting
2026-03-14 09:56:26.788106 | controller |       test step 0-cleanup-watcher'
2026-03-14 09:56:26.788111 | controller |     - '    logger.go:42: 09:47:22 | watcher-notification/0-cleanup-watcher | test step
2026-03-14 09:56:26.788115 | controller |       completed 0-cleanup-watcher'
2026-03-14 09:56:26.788119 | controller |     - '    logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788123 | controller |       starting test step 1-deploy-with-notification'
2026-03-14 09:56:26.788128 | controller |     - '    logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788132 | controller |       Watcher:watcher-kuttl-default/watcher-kuttl created'
2026-03-14 09:56:26.788136 | controller |     - '    logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788140 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.788145 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.788149 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.788155 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.788159 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.788164 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.788168 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.788172 | controller |       == 1 ]'
2026-03-14 09:56:26.788176 | controller |     - '        else'
2026-03-14 09:56:26.788180 | controller |     - '          exit 1'
2026-03-14 09:56:26.788185 | controller |     - '        fi'
2026-03-14 09:56:26.788189 | controller |     - '        ]'
2026-03-14 09:56:26.788193 | controller |     - '    logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788197 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.788201 | controller |     - '    logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788206 | controller |       Now using project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.788210 | controller |     - '    logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788214 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.788218 | controller |     - '    logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788222 | controller |       ++ head -1'
2026-03-14 09:56:26.788227 | controller |     - '    logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788231 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.788235 | controller |     - '    logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788239 | controller |       + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.788243 | controller |     - '    logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788247 | controller |       + ''['' -n watcher-kuttl-api-0 '']'''
2026-03-14 09:56:26.788252 | controller |     - '    logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788256 | controller |       ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='''
2026-03-14 09:56:26.788262 | controller |     - '    logger.go:42: 09:47:22 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788266 | controller |       +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.788270 | controller |     - '    logger.go:42: 09:47:23 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788275 | controller |       error: Internal error occurred: error executing command in container: container
2026-03-14 09:56:26.788279 | controller |       is not created or running'
2026-03-14 09:56:26.788283 | controller |     - '    logger.go:42: 09:47:23 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788287 | controller |       ++ echo'
2026-03-14 09:56:26.788291 | controller |     - '    logger.go:42: 09:47:23 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788296 | controller |       + ''['' 0 == 1 '']'''
2026-03-14 09:56:26.788300 | controller |     - '    logger.go:42: 09:47:24 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788304 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.788308 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.788313 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.788317 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.788321 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.788325 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.788330 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.788334 | controller |       == 1 ]'
2026-03-14 09:56:26.788338 | controller |     - '        else'
2026-03-14 09:56:26.788342 | controller |     - '          exit 1'
2026-03-14 09:56:26.788346 | controller |     - '        fi'
2026-03-14 09:56:26.788351 | controller |     - '        ]'
2026-03-14 09:56:26.788355 | controller |     - '    logger.go:42: 09:47:24 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788359 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.788363 | controller |     - '    logger.go:42: 09:47:25 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788367 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.788372 | controller |     - '    logger.go:42: 09:47:25 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788376 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.788386 | controller |     - '    logger.go:42: 09:47:25 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788390 | controller |       ++ head -1'
2026-03-14 09:56:26.788394 | controller |     - '    logger.go:42: 09:47:25 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788399 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.788403 | controller |     - '    logger.go:42: 09:47:25 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788407 | controller |       + APIPOD='
2026-03-14 09:56:26.788411 | controller |     - '    logger.go:42: 09:47:26 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788416 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.788420 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.788424 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.788428 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.788433 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.788437 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.788441 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.788445 | controller |       == 1 ]'
2026-03-14 09:56:26.788449 | controller |     - '        else'
2026-03-14 09:56:26.788456 | controller |     - '          exit 1'
2026-03-14 09:56:26.788460 | controller |     - '        fi'
2026-03-14 09:56:26.788464 | controller |     - '        ]'
2026-03-14 09:56:26.788468 | controller |     - '    logger.go:42: 09:47:26 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788473 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.788477 | controller |     - '    logger.go:42: 09:47:26 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788481 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.788485 | controller |     - '    logger.go:42: 09:47:26 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788490 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.788494 | controller |     - '    logger.go:42: 09:47:26 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788498 | controller |       ++ head -1'
2026-03-14 09:56:26.788502 | controller |     - '    logger.go:42: 09:47:26 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788508 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.788512 | controller |     - '    logger.go:42: 09:47:26 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788517 | controller |       + APIPOD='
2026-03-14 09:56:26.788521 | controller |     - '    logger.go:42: 09:47:27 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788525 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.788529 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.788533 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.788538 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.788542 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.788546 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.788550 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.788554 | controller |       == 1 ]'
2026-03-14 09:56:26.788559 | controller |     - '        else'
2026-03-14 09:56:26.788563 | controller |     - '          exit 1'
2026-03-14 09:56:26.788567 | controller |     - '        fi'
2026-03-14 09:56:26.788571 | controller |     - '        ]'
2026-03-14 09:56:26.788575 | controller |     - '    logger.go:42: 09:47:27 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788580 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.788584 | controller |     - '    logger.go:42: 09:47:27 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788588 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.788592 | controller |     - '    logger.go:42: 09:47:27 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788613 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.788620 | controller |     - '    logger.go:42: 09:47:27 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788624 | controller |       ++ head -1'
2026-03-14 09:56:26.788628 | controller |     - '    logger.go:42: 09:47:27 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788632 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.788637 | controller |     - '    logger.go:42: 09:47:28 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788641 | controller |       + APIPOD='
2026-03-14 09:56:26.788645 | controller |     - '    logger.go:42: 09:47:29 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788649 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.788654 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.788658 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.788662 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.788666 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.788671 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.788677 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.788681 | controller |       == 1 ]'
2026-03-14 09:56:26.788685 | controller |     - '        else'
2026-03-14 09:56:26.788689 | controller |     - '          exit 1'
2026-03-14 09:56:26.788694 | controller |     - '        fi'
2026-03-14 09:56:26.788698 | controller |     - '        ]'
2026-03-14 09:56:26.788702 | controller |     - '    logger.go:42: 09:47:29 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788713 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.788717 | controller |     - '    logger.go:42: 09:47:29 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788722 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.788726 | controller |     - '    logger.go:42: 09:47:29 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788730 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.788734 | controller |     - '    logger.go:42: 09:47:29 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788739 | controller |       ++ head -1'
2026-03-14 09:56:26.788743 | controller |     - '    logger.go:42: 09:47:29 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788747 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.788751 | controller |     - '    logger.go:42: 09:47:29 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788755 | controller |       + APIPOD='
2026-03-14 09:56:26.788760 | controller |     - '    logger.go:42: 09:47:30 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788764 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.788768 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.788772 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.788777 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.788783 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.788787 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.788791 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.788796 | controller |       == 1 ]'
2026-03-14 09:56:26.788800 | controller |     - '        else'
2026-03-14 09:56:26.788804 | controller |     - '          exit 1'
2026-03-14 09:56:26.788808 | controller |     - '        fi'
2026-03-14 09:56:26.788812 | controller |     - '        ]'
2026-03-14 09:56:26.788817 | controller |     - '    logger.go:42: 09:47:30 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788821 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.788825 | controller |     - '    logger.go:42: 09:47:30 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788829 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.788834 | controller |     - '    logger.go:42: 09:47:30 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788838 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.788842 | controller |     - '    logger.go:42: 09:47:30 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788846 | controller |       ++ head -1'
2026-03-14 09:56:26.788850 | controller |     - '    logger.go:42: 09:47:30 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788855 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.788859 | controller |     - '    logger.go:42: 09:47:30 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788863 | controller |       + APIPOD='
2026-03-14 09:56:26.788867 | controller |     - '    logger.go:42: 09:47:31 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788871 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.788876 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.788881 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.788886 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.788890 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.788894 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.788899 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.788903 | controller |       == 1 ]'
2026-03-14 09:56:26.788907 | controller |     - '        else'
2026-03-14 09:56:26.788911 | controller |     - '          exit 1'
2026-03-14 09:56:26.788915 | controller |     - '        fi'
2026-03-14 09:56:26.788920 | controller |     - '        ]'
2026-03-14 09:56:26.788924 | controller |     - '    logger.go:42: 09:47:31 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788930 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.788934 | controller |     - '    logger.go:42: 09:47:31 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788938 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.788943 | controller |     - '    logger.go:42: 09:47:31 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788947 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.788951 | controller |     - '    logger.go:42: 09:47:31 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788955 | controller |       ++ head -1'
2026-03-14 09:56:26.788959 | controller |     - '    logger.go:42: 09:47:31 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788964 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.788969 | controller |     - '    logger.go:42: 09:47:32 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788974 | controller |       + APIPOD='
2026-03-14 09:56:26.788978 | controller |     - '    logger.go:42: 09:47:33 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.788982 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.788986 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.788991 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.788995 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.788999 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.789009 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.789014 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.789018 | controller |       == 1 ]'
2026-03-14 09:56:26.789022 | controller |     - '        else'
2026-03-14 09:56:26.789027 | controller |     - '          exit 1'
2026-03-14 09:56:26.789031 | controller |     - '        fi'
2026-03-14 09:56:26.789035 | controller |     - '        ]'
2026-03-14 09:56:26.789039 | controller |     - '    logger.go:42: 09:47:33 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789044 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.789048 | controller |     - '    logger.go:42: 09:47:33 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789052 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.789056 | controller |     - '    logger.go:42: 09:47:33 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789060 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.789067 | controller |     - '    logger.go:42: 09:47:33 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789071 | controller |       ++ head -1'
2026-03-14 09:56:26.789075 | controller |     - '    logger.go:42: 09:47:33 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789079 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.789084 | controller |     - '    logger.go:42: 09:47:33 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789090 | controller |       + APIPOD='
2026-03-14 09:56:26.789094 | controller |     - '    logger.go:42: 09:47:34 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789098 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.789102 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.789107 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.789111 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.789115 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.789119 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.789123 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.789128 | controller |       == 1 ]'
2026-03-14 09:56:26.789132 | controller |     - '        else'
2026-03-14 09:56:26.789136 | controller |     - '          exit 1'
2026-03-14 09:56:26.789140 | controller |     - '        fi'
2026-03-14 09:56:26.789145 | controller |     - '        ]'
2026-03-14 09:56:26.789149 | controller |     - '    logger.go:42: 09:47:34 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789153 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.789157 | controller |     - '    logger.go:42: 09:47:34 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789162 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.789166 | controller |     - '    logger.go:42: 09:47:34 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789170 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.789174 | controller |     - '    logger.go:42: 09:47:34 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789178 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.789183 | controller |     - '    logger.go:42: 09:47:34 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789187 | controller |       ++ head -1'
2026-03-14 09:56:26.789191 | controller |     - '    logger.go:42: 09:47:34 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789195 | controller |       + APIPOD='
2026-03-14 09:56:26.789199 | controller |     - '    logger.go:42: 09:47:35 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789204 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.789208 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.789212 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.789216 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.789220 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.789225 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.789229 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.789233 | controller |       == 1 ]'
2026-03-14 09:56:26.789237 | controller |     - '        else'
2026-03-14 09:56:26.789242 | controller |     - '          exit 1'
2026-03-14 09:56:26.789246 | controller |     - '        fi'
2026-03-14 09:56:26.789250 | controller |     - '        ]'
2026-03-14 09:56:26.789254 | controller |     - '    logger.go:42: 09:47:35 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789258 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.789263 | controller |     - '    logger.go:42: 09:47:35 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789267 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.789271 | controller |     - '    logger.go:42: 09:47:35 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789275 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.789280 | controller |     - '    logger.go:42: 09:47:35 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789287 | controller |       ++ head -1'
2026-03-14 09:56:26.789291 | controller |     - '    logger.go:42: 09:47:35 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789295 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.789300 | controller |     - '    logger.go:42: 09:47:36 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789304 | controller |       + APIPOD='
2026-03-14 09:56:26.789314 | controller |     - '    logger.go:42: 09:47:37 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789318 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.789323 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.789327 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.789331 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.789336 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.789340 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.789344 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.789348 | controller |       == 1 ]'
2026-03-14 09:56:26.789352 | controller |     - '        else'
2026-03-14 09:56:26.789357 | controller |     - '          exit 1'
2026-03-14 09:56:26.789361 | controller |     - '        fi'
2026-03-14 09:56:26.789365 | controller |     - '        ]'
2026-03-14 09:56:26.789370 | controller |     - '    logger.go:42: 09:47:37 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789374 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.789378 | controller |     - '    logger.go:42: 09:47:37 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789382 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.789386 | controller |     - '    logger.go:42: 09:47:37 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789391 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.789395 | controller |     - '    logger.go:42: 09:47:37 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789399 | controller |       ++ head -1'
2026-03-14 09:56:26.789403 | controller |     - '    logger.go:42: 09:47:37 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789407 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.789412 | controller |     - '    logger.go:42: 09:47:37 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789416 | controller |       + APIPOD='
2026-03-14 09:56:26.789420 | controller |     - '    logger.go:42: 09:47:38 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789425 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.789429 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.789433 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.789437 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.789441 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.789446 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.789450 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.789454 | controller |       == 1 ]'
2026-03-14 09:56:26.789458 | controller |     - '        else'
2026-03-14 09:56:26.789463 | controller |     - '          exit 1'
2026-03-14 09:56:26.789467 | controller |     - '        fi'
2026-03-14 09:56:26.789471 | controller |     - '        ]'
2026-03-14 09:56:26.789475 | controller |     - '    logger.go:42: 09:47:38 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789480 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.789484 | controller |     - '    logger.go:42: 09:47:38 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789488 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.789495 | controller |     - '    logger.go:42: 09:47:38 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789499 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.789504 | controller |     - '    logger.go:42: 09:47:38 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789508 | controller |       ++ head -1'
2026-03-14 09:56:26.789512 | controller |     - '    logger.go:42: 09:47:38 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789516 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.789521 | controller |     - '    logger.go:42: 09:47:38 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789525 | controller |       + APIPOD='
2026-03-14 09:56:26.789529 | controller |     - '    logger.go:42: 09:47:39 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789534 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.789538 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.789542 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.789546 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.789550 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.789555 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.789559 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.789563 | controller |       == 1 ]'
2026-03-14 09:56:26.789567 | controller |     - '        else'
2026-03-14 09:56:26.789572 | controller |     - '          exit 1'
2026-03-14 09:56:26.789576 | controller |     - '        fi'
2026-03-14 09:56:26.789580 | controller |     - '        ]'
2026-03-14 09:56:26.789585 | controller |     - '    logger.go:42: 09:47:39 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789589 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.789603 | controller |     - '    logger.go:42: 09:47:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789610 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.789614 | controller |     - '    logger.go:42: 09:47:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789618 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.789629 | controller |     - '    logger.go:42: 09:47:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789633 | controller |       ++ head -1'
2026-03-14 09:56:26.789637 | controller |     - '    logger.go:42: 09:47:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789642 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.789646 | controller |     - '    logger.go:42: 09:47:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789650 | controller |       + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.789654 | controller |     - '    logger.go:42: 09:47:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789659 | controller |       + ''['' -n watcher-kuttl-api-0 '']'''
2026-03-14 09:56:26.789663 | controller |     - '    logger.go:42: 09:47:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789670 | controller |       ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='''
2026-03-14 09:56:26.789675 | controller |     - '    logger.go:42: 09:47:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789680 | controller |       +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.789684 | controller |     - '    logger.go:42: 09:47:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789688 | controller |       error: unable to upgrade connection: container not found ("watcher-api")'
2026-03-14 09:56:26.789692 | controller |     - '    logger.go:42: 09:47:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789696 | controller |       ++ echo'
2026-03-14 09:56:26.789702 | controller |     - '    logger.go:42: 09:47:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789707 | controller |       + ''['' 0 == 1 '']'''
2026-03-14 09:56:26.789711 | controller |     - '    logger.go:42: 09:47:41 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789715 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.789719 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.789724 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.789728 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.789732 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.789738 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.789742 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.789747 | controller |       == 1 ]'
2026-03-14 09:56:26.789751 | controller |     - '        else'
2026-03-14 09:56:26.789756 | controller |     - '          exit 1'
2026-03-14 09:56:26.789760 | controller |     - '        fi'
2026-03-14 09:56:26.789764 | controller |     - '        ]'
2026-03-14 09:56:26.789768 | controller |     - '    logger.go:42: 09:47:41 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789773 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.789777 | controller |     - '    logger.go:42: 09:47:41 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789781 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.789786 | controller |     - '    logger.go:42: 09:47:41 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789790 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.789794 | controller |     - '    logger.go:42: 09:47:41 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789799 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.789803 | controller |     - '    logger.go:42: 09:47:41 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789807 | controller |       ++ head -1'
2026-03-14 09:56:26.789811 | controller |     - '    logger.go:42: 09:47:41 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789816 | controller |       + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.789820 | controller |     - '    logger.go:42: 09:47:41 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789824 | controller |       + ''['' -n watcher-kuttl-api-0 '']'''
2026-03-14 09:56:26.789829 | controller |     - '    logger.go:42: 09:47:41 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789833 | controller |       ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='''
2026-03-14 09:56:26.789837 | controller |     - '    logger.go:42: 09:47:41 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789841 | controller |       +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.789846 | controller |     - '    logger.go:42: 09:47:42 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789850 | controller |       ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.789855 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.789859 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.789864 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_b745:b0f48e43119d15c3491ae912affe1d4b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.789869 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.789879 | controller |       driver = messagingv2 transport_url = ''rabbit://default_user_KcunVqmb0RRrZ6hA1Rz:BEDiv_wgPdFx5S0HWqhob2_UqEzAdLGa@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.789885 | controller |       ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.789890 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.789894 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:56:26.789898 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:56:26.789903 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.789907 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.789911 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.789915 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.789920 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.789924 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.789928 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.789932 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.789937 | controller |       tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:56:26.789941 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]''
2026-03-14 09:56:26.789945 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.789949 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.789954 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.789958 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.789962 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.789966 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.789971 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.789975 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.789979 | controller |     - '    logger.go:42: 09:47:42 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789985 | controller |       + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.789989 | controller |     - '    logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.789994 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.789998 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.790002 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.790006 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.790010 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.790015 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.790019 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.790023 | controller |       == 1 ]'
2026-03-14 09:56:26.790028 | controller |     - '        else'
2026-03-14 09:56:26.790033 | controller |     - '          exit 1'
2026-03-14 09:56:26.790037 | controller |     - '        fi'
2026-03-14 09:56:26.790042 | controller |     - '        ]'
2026-03-14 09:56:26.790046 | controller |     - '    logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790050 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.790055 | controller |     - '    logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790059 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.790063 | controller |     - '    logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790067 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.790072 | controller |     - '    logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790076 | controller |       ++ head -1'
2026-03-14 09:56:26.790080 | controller |     - '    logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790084 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.790088 | controller |     - '    logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790093 | controller |       + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.790097 | controller |     - '    logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790107 | controller |       + ''['' -n watcher-kuttl-api-0 '']'''
2026-03-14 09:56:26.790111 | controller |     - '    logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790116 | controller |       ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='''
2026-03-14 09:56:26.790120 | controller |     - '    logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790124 | controller |       +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.790129 | controller |     - '    logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790133 | controller |       ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.790137 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.790141 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.790146 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_b745:b0f48e43119d15c3491ae912affe1d4b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.790150 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.790154 | controller |       driver = messagingv2 transport_url = ''rabbit://default_user_KcunVqmb0RRrZ6hA1Rz:BEDiv_wgPdFx5S0HWqhob2_UqEzAdLGa@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.790160 | controller |       ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.790165 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.790169 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:56:26.790173 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:56:26.790178 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.790182 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.790186 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.790192 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.790196 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.790200 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.790205 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.790209 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.790213 | controller |       tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:56:26.790217 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]''
2026-03-14 09:56:26.790222 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.790226 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.790230 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.790234 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.790238 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.790243 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.790247 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.790251 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.790255 | controller |     - '    logger.go:42: 09:47:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790260 | controller |       + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.790264 | controller |     - '    logger.go:42: 09:47:44 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790268 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.790272 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.790276 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.790281 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.790285 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.790295 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.790301 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.790306 | controller |       == 1 ]'
2026-03-14 09:56:26.790310 | controller |     - '        else'
2026-03-14 09:56:26.790314 | controller |     - '          exit 1'
2026-03-14 09:56:26.790318 | controller |     - '        fi'
2026-03-14 09:56:26.790322 | controller |     - '        ]'
2026-03-14 09:56:26.790327 | controller |     - '    logger.go:42: 09:47:44 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790331 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.790335 | controller |     - '    logger.go:42: 09:47:44 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790340 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.790346 | controller |     - '    logger.go:42: 09:47:44 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790350 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.790354 | controller |     - '    logger.go:42: 09:47:44 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790359 | controller |       ++ head -1'
2026-03-14 09:56:26.790363 | controller |     - '    logger.go:42: 09:47:44 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790369 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.790375 | controller |     - '    logger.go:42: 09:47:45 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790379 | controller |       + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.790384 | controller |     - '    logger.go:42: 09:47:45 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790388 | controller |       + ''['' -n watcher-kuttl-api-0 '']'''
2026-03-14 09:56:26.790392 | controller |     - '    logger.go:42: 09:47:45 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790396 | controller |       ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='''
2026-03-14 09:56:26.790400 | controller |     - '    logger.go:42: 09:47:45 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790405 | controller |       +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.790409 | controller |     - '    logger.go:42: 09:47:45 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790413 | controller |       ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.790417 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.790422 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.790426 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_b745:b0f48e43119d15c3491ae912affe1d4b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.790430 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.790434 | controller |       driver = messagingv2 transport_url = ''rabbit://default_user_KcunVqmb0RRrZ6hA1Rz:BEDiv_wgPdFx5S0HWqhob2_UqEzAdLGa@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.790439 | controller |       ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.790443 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.790447 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:56:26.790451 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:56:26.790457 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.790462 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.790466 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.790470 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.790474 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.790479 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.790483 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.790487 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.790491 | controller |       tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:56:26.790496 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]''
2026-03-14 09:56:26.790500 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.790510 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.790517 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.790522 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.790526 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.790530 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.790534 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.790539 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.790543 | controller |     - '    logger.go:42: 09:47:45 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790547 | controller |       + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.790551 | controller |     - '    logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790555 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.790560 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.790564 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.790568 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.790572 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.790577 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.790581 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.790585 | controller |       == 1 ]'
2026-03-14 09:56:26.790589 | controller |     - '        else'
2026-03-14 09:56:26.790604 | controller |     - '          exit 1'
2026-03-14 09:56:26.790610 | controller |     - '        fi'
2026-03-14 09:56:26.790614 | controller |     - '        ]'
2026-03-14 09:56:26.790619 | controller |     - '    logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790623 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.790627 | controller |     - '    logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790631 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.790636 | controller |     - '    logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790640 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.790644 | controller |     - '    logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790648 | controller |       ++ head -1'
2026-03-14 09:56:26.790653 | controller |     - '    logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790657 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.790661 | controller |     - '    logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790665 | controller |       + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.790670 | controller |     - '    logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790676 | controller |       + ''['' -n watcher-kuttl-api-0 '']'''
2026-03-14 09:56:26.790680 | controller |     - '    logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790684 | controller |       ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='''
2026-03-14 09:56:26.790689 | controller |     - '    logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790693 | controller |       +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.790697 | controller |     - '    logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790701 | controller |       ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.790707 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.790712 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.790716 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_b745:b0f48e43119d15c3491ae912affe1d4b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.790720 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.790724 | controller |       driver = messagingv2 transport_url = ''rabbit://default_user_KcunVqmb0RRrZ6hA1Rz:BEDiv_wgPdFx5S0HWqhob2_UqEzAdLGa@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.790729 | controller |       ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.790733 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.790737 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:56:26.790748 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:56:26.790753 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.790757 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.790761 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.790765 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.790770 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.790774 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.790778 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.790782 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.790787 | controller |       tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:56:26.790791 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]''
2026-03-14 09:56:26.790795 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.790799 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.790803 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.790808 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.790812 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.790816 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.790820 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.790824 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.790829 | controller |     - '    logger.go:42: 09:47:46 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790833 | controller |       + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.790837 | controller |     - '    logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790843 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.790847 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.790851 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.790855 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.790860 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.790864 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.790868 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.790872 | controller |       == 1 ]'
2026-03-14 09:56:26.790877 | controller |     - '        else'
2026-03-14 09:56:26.790881 | controller |     - '          exit 1'
2026-03-14 09:56:26.790885 | controller |     - '        fi'
2026-03-14 09:56:26.790889 | controller |     - '        ]'
2026-03-14 09:56:26.790893 | controller |     - '    logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790898 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.790902 | controller |     - '    logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790906 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.790910 | controller |     - '    logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790915 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.790921 | controller |     - '    logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790925 | controller |       ++ head -1'
2026-03-14 09:56:26.790929 | controller |     - '    logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790934 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.790938 | controller |     - '    logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790942 | controller |       + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.790946 | controller |     - '    logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790951 | controller |       + ''['' -n watcher-kuttl-api-0 '']'''
2026-03-14 09:56:26.790955 | controller |     - '    logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790959 | controller |       ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='''
2026-03-14 09:56:26.790963 | controller |     - '    logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790968 | controller |       +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.790972 | controller |     - '    logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.790982 | controller |       ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.790987 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.790991 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.790995 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_b745:b0f48e43119d15c3491ae912affe1d4b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.791000 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.791004 | controller |       driver = messagingv2 transport_url = ''rabbit://default_user_KcunVqmb0RRrZ6hA1Rz:BEDiv_wgPdFx5S0HWqhob2_UqEzAdLGa@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.791009 | controller |       ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.791022 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.791027 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:56:26.791053 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:56:26.791057 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.791061 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.791066 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.791070 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.791074 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.791079 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.791083 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.791087 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.791092 | controller |       tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:56:26.791096 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]''
2026-03-14 09:56:26.791100 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.791104 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.791109 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.791113 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.791117 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.791122 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.791126 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.791130 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.791135 | controller |     - '    logger.go:42: 09:47:48 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791139 | controller |       + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.791143 | controller |     - '    logger.go:42: 09:47:49 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791148 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.791152 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.791156 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.791160 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.791165 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.791169 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.791173 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.791178 | controller |       == 1 ]'
2026-03-14 09:56:26.791182 | controller |     - '        else'
2026-03-14 09:56:26.791186 | controller |     - '          exit 1'
2026-03-14 09:56:26.791191 | controller |     - '        fi'
2026-03-14 09:56:26.791195 | controller |     - '        ]'
2026-03-14 09:56:26.791199 | controller |     - '    logger.go:42: 09:47:49 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791204 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.791210 | controller |     - '    logger.go:42: 09:47:49 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791221 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.791226 | controller |     - '    logger.go:42: 09:47:49 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791230 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.791235 | controller |     - '    logger.go:42: 09:47:49 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791239 | controller |       ++ head -1'
2026-03-14 09:56:26.791243 | controller |     - '    logger.go:42: 09:47:49 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791248 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.791252 | controller |     - '    logger.go:42: 09:47:49 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791256 | controller |       + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.791260 | controller |     - '    logger.go:42: 09:47:49 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791265 | controller |       + ''['' -n watcher-kuttl-api-0 '']'''
2026-03-14 09:56:26.791269 | controller |     - '    logger.go:42: 09:47:49 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791273 | controller |       ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='''
2026-03-14 09:56:26.791278 | controller |     - '    logger.go:42: 09:47:49 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791282 | controller |       +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.791286 | controller |     - '    logger.go:42: 09:47:50 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791290 | controller |       ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.791295 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.791299 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.791303 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_b745:b0f48e43119d15c3491ae912affe1d4b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.791308 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.791312 | controller |       driver = messagingv2 transport_url = ''rabbit://default_user_KcunVqmb0RRrZ6hA1Rz:BEDiv_wgPdFx5S0HWqhob2_UqEzAdLGa@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.791318 | controller |       ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.791324 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.791330 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:56:26.791335 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:56:26.791341 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.791346 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.791352 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.791358 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.791364 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.791369 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.791378 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.791384 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.791392 | controller |       tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:56:26.791397 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]''
2026-03-14 09:56:26.791403 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.791409 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.791414 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.791419 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.791425 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.791430 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.791445 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.791453 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.791459 | controller |     - '    logger.go:42: 09:47:50 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791465 | controller |       + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.791470 | controller |     - '    logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791476 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.791482 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.791488 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.791493 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.791507 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.791512 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.791518 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.791523 | controller |       == 1 ]'
2026-03-14 09:56:26.791529 | controller |     - '        else'
2026-03-14 09:56:26.791535 | controller |     - '          exit 1'
2026-03-14 09:56:26.791541 | controller |     - '        fi'
2026-03-14 09:56:26.791546 | controller |     - '        ]'
2026-03-14 09:56:26.791552 | controller |     - '    logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791557 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.791563 | controller |     - '    logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791569 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.791575 | controller |     - '    logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791580 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.791586 | controller |     - '    logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791591 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.791613 | controller |     - '    logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791618 | controller |       ++ head -1'
2026-03-14 09:56:26.791622 | controller |     - '    logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791627 | controller |       + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.791631 | controller |     - '    logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791638 | controller |       + ''['' -n watcher-kuttl-api-0 '']'''
2026-03-14 09:56:26.791642 | controller |     - '    logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791647 | controller |       ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='''
2026-03-14 09:56:26.791651 | controller |     - '    logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791655 | controller |       +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.791662 | controller |     - '    logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791666 | controller |       ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.791672 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.791676 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.791680 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_b745:b0f48e43119d15c3491ae912affe1d4b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.791685 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.791689 | controller |       driver = messagingv2 transport_url = ''rabbit://default_user_KcunVqmb0RRrZ6hA1Rz:BEDiv_wgPdFx5S0HWqhob2_UqEzAdLGa@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.791693 | controller |       ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.791697 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.791702 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:56:26.791706 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:56:26.791710 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.791714 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.791718 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.791723 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.791735 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.791740 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.791744 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.791748 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.791753 | controller |       tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:56:26.791757 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]''
2026-03-14 09:56:26.791761 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.791765 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.791769 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.791774 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.791781 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.791785 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.791789 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.791794 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.791800 | controller |     - '    logger.go:42: 09:47:51 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791804 | controller |       + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.791808 | controller |     - '    logger.go:42: 09:47:52 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791812 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.791817 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.791821 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.791825 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.791829 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.791834 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.791838 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:56:26.791842 | controller |       == 1 ]'
2026-03-14 09:56:26.791846 | controller |     - '        else'
2026-03-14 09:56:26.791851 | controller |     - '          exit 1'
2026-03-14 09:56:26.791855 | controller |     - '        fi'
2026-03-14 09:56:26.791859 | controller |     - '        ]'
2026-03-14 09:56:26.791863 | controller |     - '    logger.go:42: 09:47:52 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791868 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:56:26.791872 | controller |     - '    logger.go:42: 09:47:52 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791876 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.791880 | controller |     - '    logger.go:42: 09:47:53 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791885 | controller |       ++ grep -v ''^$'''
2026-03-14 09:56:26.791889 | controller |     - '    logger.go:42: 09:47:53 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791893 | controller |       ++ head -1'
2026-03-14 09:56:26.791898 | controller |     - '    logger.go:42: 09:47:53 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791902 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.791906 | controller |     - '    logger.go:42: 09:47:53 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791910 | controller |       + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.791915 | controller |     - '    logger.go:42: 09:47:53 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791919 | controller |       + ''['' -n watcher-kuttl-api-0 '']'''
2026-03-14 09:56:26.791923 | controller |     - '    logger.go:42: 09:47:53 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791927 | controller |       ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='''
2026-03-14 09:56:26.791932 | controller |     - '    logger.go:42: 09:47:53 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791936 | controller |       +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.791940 | controller |     - '    logger.go:42: 09:47:53 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.791945 | controller |       ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.791949 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.791955 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.791966 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_b745:b0f48e43119d15c3491ae912affe1d4b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.791971 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.791975 | controller |       driver = messagingv2 transport_url = ''rabbit://default_user_KcunVqmb0RRrZ6hA1Rz:BEDiv_wgPdFx5S0HWqhob2_UqEzAdLGa@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.791979 | controller |       ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.791984 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.791988 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:56:26.791993 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:56:26.791997 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.792001 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.792006 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.792010 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.792014 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.792018 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.792023 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.792027 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.792031 | controller |       tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:56:26.792036 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]''
2026-03-14 09:56:26.792040 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.792044 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.792049 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.792053 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.792057 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.792061 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.792066 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.792070 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.792074 | controller |     - '    logger.go:42: 09:47:53 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.792078 | controller |       + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.792083 | controller |     - '    logger.go:42: 09:47:53 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:56:26.792087 | controller |       test step completed 1-deploy-with-notification'
2026-03-14 09:56:26.792091 | controller |     - '    logger.go:42: 09:47:53 | watcher-notification/2-cleanup-watcher | starting
2026-03-14 09:56:26.792096 | controller |       test step 2-cleanup-watcher'
2026-03-14 09:56:26.792100 | controller |     - '    logger.go:42: 09:47:59 | watcher-notification/2-cleanup-watcher | test step
2026-03-14 09:56:26.792106 | controller |       completed 2-cleanup-watcher'
2026-03-14 09:56:26.792110 | controller |     - '    logger.go:42: 09:47:59 | watcher-notification | skipping kubernetes event logging'
2026-03-14 09:56:26.792116 | controller |     - === CONT  kuttl/harness/watcher-topology
2026-03-14 09:56:26.792121 | controller |     - '    logger.go:42: 09:47:59 | watcher-topology | Skipping creation of user-supplied
2026-03-14 09:56:26.792125 | controller |       namespace: watcher-kuttl-default'
2026-03-14 09:56:26.792129 | controller |     - '    logger.go:42: 09:47:59 | watcher-topology/0-cleanup-watcher | starting test
2026-03-14 09:56:26.792134 | controller |       step 0-cleanup-watcher'
2026-03-14 09:56:26.792138 | controller |     - '    logger.go:42: 09:47:59 | watcher-topology/0-cleanup-watcher | test step completed
2026-03-14 09:56:26.792142 | controller |       0-cleanup-watcher'
2026-03-14 09:56:26.792147 | controller |     - '    logger.go:42: 09:47:59 | watcher-topology/1-deploy-with-topology | starting
2026-03-14 09:56:26.792151 | controller |       test step 1-deploy-with-topology'
2026-03-14 09:56:26.792155 | controller |     - '    logger.go:42: 09:47:59 | watcher-topology/1-deploy-with-topology | Topology:watcher-kuttl-default/watcher-api
2026-03-14 09:56:26.792159 | controller |       created'
2026-03-14 09:56:26.792164 | controller |     - '    logger.go:42: 09:47:59 | watcher-topology/1-deploy-with-topology | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:56:26.792168 | controller |       created'
2026-03-14 09:56:26.792172 | controller |     - '    logger.go:42: 09:48:24 | watcher-topology/1-deploy-with-topology | test step
2026-03-14 09:56:26.792183 | controller |       completed 1-deploy-with-topology'
2026-03-14 09:56:26.792188 | controller |     - '    logger.go:42: 09:48:24 | watcher-topology/2-cleanup-watcher | starting test
2026-03-14 09:56:26.792192 | controller |       step 2-cleanup-watcher'
2026-03-14 09:56:26.792196 | controller |     - '    logger.go:42: 09:48:24 | watcher-topology/2-cleanup-watcher | running command:
2026-03-14 09:56:26.792201 | controller |       [sh -c set -ex'
2026-03-14 09:56:26.792205 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.792209 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:56:26.792214 | controller |     - '        ]'
2026-03-14 09:56:26.792218 | controller |     - '    logger.go:42: 09:48:24 | watcher-topology/2-cleanup-watcher | + oc exec -n
2026-03-14 09:56:26.792222 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.792226 | controller |       -c Type'
2026-03-14 09:56:26.792231 | controller |     - '    logger.go:42: 09:48:24 | watcher-topology/2-cleanup-watcher | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.792235 | controller |     - '    logger.go:42: 09:48:27 | watcher-topology/2-cleanup-watcher | + ''['' 1 ==
2026-03-14 09:56:26.792239 | controller |       0 '']'''
2026-03-14 09:56:26.792244 | controller |     - '    logger.go:42: 09:48:28 | watcher-topology/2-cleanup-watcher | running command:
2026-03-14 09:56:26.792248 | controller |       [sh -c set -ex'
2026-03-14 09:56:26.792252 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.792256 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:56:26.792261 | controller |     - '        ]'
2026-03-14 09:56:26.792265 | controller |     - '    logger.go:42: 09:48:28 | watcher-topology/2-cleanup-watcher | + oc exec -n
2026-03-14 09:56:26.792269 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.792274 | controller |       -c Type'
2026-03-14 09:56:26.792278 | controller |     - '    logger.go:42: 09:48:28 | watcher-topology/2-cleanup-watcher | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.792282 | controller |     - '    logger.go:42: 09:48:30 | watcher-topology/2-cleanup-watcher | + ''['' 0 ==
2026-03-14 09:56:26.792286 | controller |       0 '']'''
2026-03-14 09:56:26.792291 | controller |     - '    logger.go:42: 09:48:31 | watcher-topology/2-cleanup-watcher | running command:
2026-03-14 09:56:26.792295 | controller |       [sh -c set -ex'
2026-03-14 09:56:26.792299 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.792306 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:56:26.792310 | controller |     - '        ]'
2026-03-14 09:56:26.792314 | controller |     - '    logger.go:42: 09:48:31 | watcher-topology/2-cleanup-watcher | + oc exec -n
2026-03-14 09:56:26.792319 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.792323 | controller |       -c Type'
2026-03-14 09:56:26.792327 | controller |     - '    logger.go:42: 09:48:31 | watcher-topology/2-cleanup-watcher | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.792332 | controller |     - '    logger.go:42: 09:48:33 | watcher-topology/2-cleanup-watcher | + ''['' 0 ==
2026-03-14 09:56:26.792336 | controller |       0 '']'''
2026-03-14 09:56:26.792340 | controller |     - '    logger.go:42: 09:48:34 | watcher-topology/2-cleanup-watcher | running command:
2026-03-14 09:56:26.792345 | controller |       [sh -c set -ex'
2026-03-14 09:56:26.792349 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.792353 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:56:26.792357 | controller |     - '        ]'
2026-03-14 09:56:26.792362 | controller |     - '    logger.go:42: 09:48:34 | watcher-topology/2-cleanup-watcher | + oc exec -n
2026-03-14 09:56:26.792366 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.792370 | controller |       -c Type'
2026-03-14 09:56:26.792375 | controller |     - '    logger.go:42: 09:48:34 | watcher-topology/2-cleanup-watcher | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.792379 | controller |     - '    logger.go:42: 09:48:37 | watcher-topology/2-cleanup-watcher | + ''['' 0 ==
2026-03-14 09:56:26.792383 | controller |       0 '']'''
2026-03-14 09:56:26.792388 | controller |     - '    logger.go:42: 09:48:37 | watcher-topology/2-cleanup-watcher | test step completed
2026-03-14 09:56:26.792392 | controller |       2-cleanup-watcher'
2026-03-14 09:56:26.792396 | controller |     - '    logger.go:42: 09:48:37 | watcher-topology | skipping kubernetes event logging'
2026-03-14 09:56:26.792400 | controller |     - === CONT  kuttl/harness/watcher-tls-certs-change
2026-03-14 09:56:26.792405 | controller |     - '    logger.go:42: 09:48:37 | watcher-tls-certs-change | Skipping creation of user-supplied
2026-03-14 09:56:26.792409 | controller |       namespace: watcher-kuttl-default'
2026-03-14 09:56:26.792413 | controller |     - '    logger.go:42: 09:48:37 | watcher-tls-certs-change/0-cleanup-watcher | starting
2026-03-14 09:56:26.792418 | controller |       test step 0-cleanup-watcher'
2026-03-14 09:56:26.792422 | controller |     - '    logger.go:42: 09:48:37 | watcher-tls-certs-change/0-cleanup-watcher | test
2026-03-14 09:56:26.792426 | controller |       step completed 0-cleanup-watcher'
2026-03-14 09:56:26.792431 | controller |     - '    logger.go:42: 09:48:37 | watcher-tls-certs-change/1-deploy-with-tlse | starting
2026-03-14 09:56:26.792437 | controller |       test step 1-deploy-with-tlse'
2026-03-14 09:56:26.792441 | controller |     - '    logger.go:42: 09:48:37 | watcher-tls-certs-change/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-internal-svc
2026-03-14 09:56:26.792447 | controller |       created'
2026-03-14 09:56:26.792451 | controller |     - '    logger.go:42: 09:48:37 | watcher-tls-certs-change/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-public-svc
2026-03-14 09:56:26.792455 | controller |       created'
2026-03-14 09:56:26.792460 | controller |     - '    logger.go:42: 09:48:37 | watcher-tls-certs-change/1-deploy-with-tlse | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:56:26.792464 | controller |       created'
2026-03-14 09:56:26.792474 | controller |     - '    logger.go:42: 09:48:37 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:56:26.792479 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.792484 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.792488 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:56:26.792492 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.792497 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.792501 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.792507 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:56:26.792511 | controller |     - '          exit 1'
2026-03-14 09:56:26.792516 | controller |     - '        fi'
2026-03-14 09:56:26.792520 | controller |     - '        '
2026-03-14 09:56:26.792524 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.792529 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.792533 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.792537 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.792541 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.792546 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:56:26.792550 | controller |     - '          exit 1'
2026-03-14 09:56:26.792554 | controller |     - '        fi'
2026-03-14 09:56:26.792559 | controller |     - '        ]'
2026-03-14 09:56:26.792563 | controller |     - '    logger.go:42: 09:48:37 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.792567 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:56:26.792572 | controller |     - '    logger.go:42: 09:48:37 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:56:26.792576 | controller |       from server (NotFound): pods "watcher-kuttl-api-0" not found'
2026-03-14 09:56:26.792580 | controller |     - '    logger.go:42: 09:48:37 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:56:26.792586 | controller |     - '    logger.go:42: 09:48:38 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:56:26.792591 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.792606 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.792612 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:56:26.792617 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.792621 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.792625 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.792630 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:56:26.792634 | controller |     - '          exit 1'
2026-03-14 09:56:26.792638 | controller |     - '        fi'
2026-03-14 09:56:26.792643 | controller |     - '        '
2026-03-14 09:56:26.792647 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.792651 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.792656 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.792660 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.792664 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.792668 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:56:26.792673 | controller |     - '          exit 1'
2026-03-14 09:56:26.792677 | controller |     - '        fi'
2026-03-14 09:56:26.792681 | controller |     - '        ]'
2026-03-14 09:56:26.792686 | controller |     - '    logger.go:42: 09:48:38 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.792690 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:56:26.792694 | controller |     - '    logger.go:42: 09:48:38 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:56:26.792698 | controller |       from server (NotFound): pods "watcher-kuttl-api-0" not found'
2026-03-14 09:56:26.792703 | controller |     - '    logger.go:42: 09:48:38 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:56:26.792707 | controller |     - '    logger.go:42: 09:48:39 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:56:26.792711 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.792717 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.792722 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:56:26.792726 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.792730 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.792735 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.792739 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:56:26.792743 | controller |     - '          exit 1'
2026-03-14 09:56:26.792748 | controller |     - '        fi'
2026-03-14 09:56:26.792752 | controller |     - '        '
2026-03-14 09:56:26.792758 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.792763 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.792769 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.792773 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.792778 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.792789 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:56:26.792794 | controller |     - '          exit 1'
2026-03-14 09:56:26.792798 | controller |     - '        fi'
2026-03-14 09:56:26.792803 | controller |     - '        ]'
2026-03-14 09:56:26.792807 | controller |     - '    logger.go:42: 09:48:39 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.792812 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:56:26.792816 | controller |     - '    logger.go:42: 09:48:40 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:56:26.792821 | controller |       from server (NotFound): pods "watcher-kuttl-api-0" not found'
2026-03-14 09:56:26.792825 | controller |     - '    logger.go:42: 09:48:40 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:56:26.792829 | controller |     - '    logger.go:42: 09:48:41 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:56:26.792834 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.792838 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.792843 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:56:26.792847 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.792852 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.792856 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.792860 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:56:26.792865 | controller |     - '          exit 1'
2026-03-14 09:56:26.792869 | controller |     - '        fi'
2026-03-14 09:56:26.792873 | controller |     - '        '
2026-03-14 09:56:26.792878 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.792882 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.792887 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.792891 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.792895 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.792900 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:56:26.792904 | controller |     - '          exit 1'
2026-03-14 09:56:26.792908 | controller |     - '        fi'
2026-03-14 09:56:26.792913 | controller |     - '        ]'
2026-03-14 09:56:26.792917 | controller |     - '    logger.go:42: 09:48:41 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.792923 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:56:26.792927 | controller |     - '    logger.go:42: 09:48:41 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:56:26.792933 | controller |       from server (NotFound): pods "watcher-kuttl-api-0" not found'
2026-03-14 09:56:26.792938 | controller |     - '    logger.go:42: 09:48:41 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:56:26.792942 | controller |     - '    logger.go:42: 09:48:42 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:56:26.792946 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.792951 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.792955 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:56:26.792959 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.792964 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.792968 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.792972 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:56:26.792980 | controller |     - '          exit 1'
2026-03-14 09:56:26.792984 | controller |     - '        fi'
2026-03-14 09:56:26.792989 | controller |     - '        '
2026-03-14 09:56:26.792993 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.792997 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.793002 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.793006 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.793010 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.793014 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:56:26.793019 | controller |     - '          exit 1'
2026-03-14 09:56:26.793023 | controller |     - '        fi'
2026-03-14 09:56:26.793027 | controller |     - '        ]'
2026-03-14 09:56:26.793032 | controller |     - '    logger.go:42: 09:48:42 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.793036 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:56:26.793040 | controller |     - '    logger.go:42: 09:48:42 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:56:26.793045 | controller |       from server (NotFound): pods "watcher-kuttl-api-0" not found'
2026-03-14 09:56:26.793049 | controller |     - '    logger.go:42: 09:48:42 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:56:26.793053 | controller |     - '    logger.go:42: 09:48:43 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:56:26.793057 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.793062 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.793066 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:56:26.793070 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.793075 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.793085 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.793089 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:56:26.793094 | controller |     - '          exit 1'
2026-03-14 09:56:26.793098 | controller |     - '        fi'
2026-03-14 09:56:26.793103 | controller |     - '        '
2026-03-14 09:56:26.793107 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.793111 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.793116 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.793120 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.793124 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.793129 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:56:26.793135 | controller |     - '          exit 1'
2026-03-14 09:56:26.793201 | controller |     - '        fi'
2026-03-14 09:56:26.793208 | controller |     - '        ]'
2026-03-14 09:56:26.793212 | controller |     - '    logger.go:42: 09:48:43 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.793217 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:56:26.793221 | controller |     - '    logger.go:42: 09:48:43 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:56:26.793225 | controller |       from server (NotFound): pods "watcher-kuttl-api-0" not found'
2026-03-14 09:56:26.793230 | controller |     - '    logger.go:42: 09:48:43 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:56:26.793234 | controller |     - '    logger.go:42: 09:48:44 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:56:26.793238 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.793243 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.793247 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:56:26.793251 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.793256 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.793260 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.793264 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:56:26.793269 | controller |     - '          exit 1'
2026-03-14 09:56:26.793273 | controller |     - '        fi'
2026-03-14 09:56:26.793277 | controller |     - '        '
2026-03-14 09:56:26.793282 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.793286 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.793290 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.793294 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.793299 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.793303 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:56:26.793307 | controller |     - '          exit 1'
2026-03-14 09:56:26.793311 | controller |     - '        fi'
2026-03-14 09:56:26.793316 | controller |     - '        ]'
2026-03-14 09:56:26.793320 | controller |     - '    logger.go:42: 09:48:44 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.793324 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:56:26.793328 | controller |     - '    logger.go:42: 09:48:45 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:56:26.793332 | controller |       from server (NotFound): pods "watcher-kuttl-api-0" not found'
2026-03-14 09:56:26.793337 | controller |     - '    logger.go:42: 09:48:45 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:56:26.793341 | controller |     - '    logger.go:42: 09:48:46 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:56:26.793345 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.793349 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.793354 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:56:26.793358 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.793362 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.793366 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.793371 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:56:26.793375 | controller |     - '          exit 1'
2026-03-14 09:56:26.793379 | controller |     - '        fi'
2026-03-14 09:56:26.793383 | controller |     - '        '
2026-03-14 09:56:26.793388 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.793394 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.793398 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.793403 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.793407 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.793411 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:56:26.793418 | controller |     - '          exit 1'
2026-03-14 09:56:26.793422 | controller |     - '        fi'
2026-03-14 09:56:26.793426 | controller |     - '        ]'
2026-03-14 09:56:26.793430 | controller |     - '    logger.go:42: 09:48:46 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.793435 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:56:26.793439 | controller |     - '    logger.go:42: 09:48:46 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:56:26.793451 | controller |       from server (NotFound): pods "watcher-kuttl-api-0" not found'
2026-03-14 09:56:26.793456 | controller |     - '    logger.go:42: 09:48:46 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:56:26.793460 | controller |     - '    logger.go:42: 09:48:47 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:56:26.793464 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.793469 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.793473 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:56:26.793477 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.793481 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.793486 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.793490 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:56:26.793494 | controller |     - '          exit 1'
2026-03-14 09:56:26.793498 | controller |     - '        fi'
2026-03-14 09:56:26.793503 | controller |     - '        '
2026-03-14 09:56:26.793507 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.793511 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.793515 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.793520 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.793524 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.793528 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:56:26.793532 | controller |     - '          exit 1'
2026-03-14 09:56:26.793537 | controller |     - '        fi'
2026-03-14 09:56:26.793541 | controller |     - '        ]'
2026-03-14 09:56:26.793545 | controller |     - '    logger.go:42: 09:48:47 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.793549 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:56:26.793554 | controller |     - '    logger.go:42: 09:48:47 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:56:26.793558 | controller |       from server (NotFound): pods "watcher-kuttl-api-0" not found'
2026-03-14 09:56:26.793562 | controller |     - '    logger.go:42: 09:48:47 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:56:26.793566 | controller |     - '    logger.go:42: 09:48:48 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:56:26.793571 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.793575 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.793579 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:56:26.793583 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.793588 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.793608 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.793615 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:56:26.793629 | controller |     - '          exit 1'
2026-03-14 09:56:26.793634 | controller |     - '        fi'
2026-03-14 09:56:26.793638 | controller |     - '        '
2026-03-14 09:56:26.793642 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.793646 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.793651 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.793655 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.793659 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.793663 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:56:26.793668 | controller |     - '          exit 1'
2026-03-14 09:56:26.793672 | controller |     - '        fi'
2026-03-14 09:56:26.793676 | controller |     - '        ]'
2026-03-14 09:56:26.793680 | controller |     - '    logger.go:42: 09:48:48 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.793685 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:56:26.793689 | controller |     - '    logger.go:42: 09:48:48 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:56:26.793693 | controller |       from server (NotFound): pods "watcher-kuttl-api-0" not found'
2026-03-14 09:56:26.793697 | controller |     - '    logger.go:42: 09:48:48 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:56:26.793702 | controller |     - '    logger.go:42: 09:48:50 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:56:26.793706 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.793710 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.793714 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:56:26.793719 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.793723 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.793727 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.793731 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:56:26.793735 | controller |     - '          exit 1'
2026-03-14 09:56:26.793740 | controller |     - '        fi'
2026-03-14 09:56:26.793744 | controller |     - '        '
2026-03-14 09:56:26.793748 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.793752 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.793764 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.793768 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.793773 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.793777 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:56:26.793781 | controller |     - '          exit 1'
2026-03-14 09:56:26.793785 | controller |     - '        fi'
2026-03-14 09:56:26.793790 | controller |     - '        ]'
2026-03-14 09:56:26.793794 | controller |     - '    logger.go:42: 09:48:50 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.793798 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:56:26.793802 | controller |     - '    logger.go:42: 09:48:50 | watcher-tls-certs-change/1-deploy-with-tlse | error:
2026-03-14 09:56:26.793807 | controller |       unable to upgrade connection: container not found ("watcher-api")'
2026-03-14 09:56:26.793811 | controller |     - '    logger.go:42: 09:48:50 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:56:26.793815 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:56:26.793822 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.793826 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.793830 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:56:26.793834 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.793839 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.793843 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.793847 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:56:26.793852 | controller |     - '          exit 1'
2026-03-14 09:56:26.793856 | controller |     - '        fi'
2026-03-14 09:56:26.793860 | controller |     - '        '
2026-03-14 09:56:26.793864 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.793869 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.793873 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.793879 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.793883 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.793888 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:56:26.793892 | controller |     - '          exit 1'
2026-03-14 09:56:26.793896 | controller |     - '        fi'
2026-03-14 09:56:26.793900 | controller |     - '        ]'
2026-03-14 09:56:26.793905 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.793909 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:56:26.793913 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN
2026-03-14 09:56:26.793917 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.793923 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.793928 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.793932 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.793937 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.793941 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.793945 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.793949 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.793954 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.793958 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.793962 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.793966 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.793970 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.793983 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.793988 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.793992 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.793996 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.794000 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.794005 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.794009 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.794013 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.794018 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.794022 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.794026 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:56:26.794031 | controller |       --decode'
2026-03-14 09:56:26.794035 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.794039 | controller |       get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:56:26.794044 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN
2026-03-14 09:56:26.794048 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.794052 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.794056 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.794063 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.794067 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.794071 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.794076 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.794080 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.794084 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.794089 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.794093 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.794097 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.794104 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.794108 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.794112 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.794117 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.794121 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.794131 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.794136 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.794140 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.794145 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.794149 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.794153 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.794160 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:56:26.794164 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.794169 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.794173 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.794177 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.794182 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.794186 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.794192 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.794197 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.794201 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.794205 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.794210 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.794214 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.794218 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.794226 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.794230 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.794234 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.794239 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.794243 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.794247 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.794252 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.794256 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.794260 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.794265 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.794269 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.794273 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.794284 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.794289 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.794293 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.794298 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.794302 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.794308 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.794313 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.794317 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.794321 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.794326 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.794330 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.794334 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.794342 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.794346 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.794350 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.794355 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.794359 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.794363 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.794367 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.794372 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:56:26.794376 | controller |     - '    logger.go:42: 09:48:51 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.794381 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:56:26.794385 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN
2026-03-14 09:56:26.794389 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.794394 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.794398 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.794402 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.794407 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.794411 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.794415 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.794425 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.794430 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.794437 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.794441 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.794445 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.794450 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.794454 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.794460 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.794464 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.794469 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.794473 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.794477 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.794482 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.794486 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.794490 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.794496 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.794501 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:56:26.794505 | controller |       --decode'
2026-03-14 09:56:26.794510 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.794514 | controller |       get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:56:26.794518 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN
2026-03-14 09:56:26.794523 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.794527 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.794531 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.794536 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.794540 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.794544 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.794549 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.794553 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.794557 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.794561 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.794566 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.794578 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.794583 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.794589 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.794605 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.794611 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.794616 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.794620 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.794625 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.794629 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.794633 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.794638 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.794642 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.794646 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:56:26.794651 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.794655 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.794659 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.794664 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.794668 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.794672 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.794677 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.794681 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.794685 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.794690 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.794694 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.794698 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.794703 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.794709 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.794715 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.794719 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.794724 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.794734 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.794739 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.794744 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.794748 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.794752 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.794757 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.794761 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.794765 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.794770 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.794774 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.794778 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.794783 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.794787 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.794791 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.794796 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.794800 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.794804 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.794808 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.794813 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.794817 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.794823 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.794829 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.794833 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.794838 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.794842 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.794846 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.794851 | controller |     - '    logger.go:42: 09:48:52 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.794855 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:56:26.794859 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:56:26.794864 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.794868 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.794872 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:56:26.794876 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.794886 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.794890 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.794895 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:56:26.794899 | controller |     - '          exit 1'
2026-03-14 09:56:26.794903 | controller |     - '        fi'
2026-03-14 09:56:26.794908 | controller |     - '        '
2026-03-14 09:56:26.794912 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.794916 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.794921 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.794925 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.794929 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.794933 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:56:26.794938 | controller |     - '          exit 1'
2026-03-14 09:56:26.794942 | controller |     - '        fi'
2026-03-14 09:56:26.794946 | controller |     - '        ]'
2026-03-14 09:56:26.794951 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.794955 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:56:26.794959 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN
2026-03-14 09:56:26.794964 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.794968 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.794972 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.794977 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.794981 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.794985 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.794991 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.794996 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.795000 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.795004 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.795009 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.795013 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.795017 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.795022 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.795026 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.795045 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.795052 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.795061 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.795066 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.795070 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.795074 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.795079 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.795090 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.795095 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:56:26.795099 | controller |       --decode'
2026-03-14 09:56:26.795104 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.795108 | controller |       get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:56:26.795112 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN
2026-03-14 09:56:26.795117 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.795121 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.795125 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.795130 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.795134 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.795140 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.795145 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.795149 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.795153 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.795158 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.795162 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.795166 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.795170 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.795175 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.795179 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.795183 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.795188 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.795192 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.795196 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.795203 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.795207 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.795211 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.795216 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.795220 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:56:26.795224 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.795229 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.795233 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.795237 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.795248 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.795252 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.795258 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.795263 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.795267 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.795271 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.795276 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.795280 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.795284 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.795288 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.795293 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.795297 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.795301 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.795305 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.795310 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.795314 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.795318 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.795323 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.795327 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.795335 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.795340 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.795346 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.795350 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.795355 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.795360 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.795365 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.795372 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.795376 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.795389 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.795393 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.795398 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.795402 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.795406 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.795411 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.795415 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.795419 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.795424 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.795428 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.795432 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.795437 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.795441 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:56:26.795445 | controller |     - '    logger.go:42: 09:48:53 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.795449 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:56:26.795454 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN
2026-03-14 09:56:26.795458 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.795462 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.795466 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.795471 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.795477 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.795481 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.795485 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.795490 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.795496 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.795500 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.795504 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.795509 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.795513 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.795517 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.795521 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.795532 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.795536 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.795541 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.795545 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.795549 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.795554 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.795558 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.795562 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.795567 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:56:26.795571 | controller |       --decode'
2026-03-14 09:56:26.795575 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.795582 | controller |       get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:56:26.795586 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN
2026-03-14 09:56:26.795591 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.795606 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.795621 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.795625 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.795630 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.795634 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.795640 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.795647 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.795651 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.795656 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.795660 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.795664 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.795668 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.795673 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.795677 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.795682 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.795686 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.795690 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.795694 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.795706 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.795711 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.795715 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.795720 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.795724 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:56:26.795728 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.795733 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.795737 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.795741 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.795746 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.795750 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.795754 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.795758 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.795774 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.795778 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.795783 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.795787 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.795791 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.795795 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.795800 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.795804 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.795808 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.795813 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.795817 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.795821 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.795855 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.795862 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.795866 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.795871 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.795875 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.795880 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.795891 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.795896 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.795900 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.795930 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.795936 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.795940 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.795949 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.795954 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.795958 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.795962 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.795967 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.795971 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.795975 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.795979 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.795984 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.795988 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.795992 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.795997 | controller |     - '    logger.go:42: 09:48:54 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.796001 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:56:26.796005 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:56:26.796010 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.796014 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.796018 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:56:26.796023 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.796027 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.796031 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.796036 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:56:26.796040 | controller |     - '          exit 1'
2026-03-14 09:56:26.796044 | controller |     - '        fi'
2026-03-14 09:56:26.796049 | controller |     - '        '
2026-03-14 09:56:26.796053 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.796057 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.796061 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.796066 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.796070 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.796074 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:56:26.796079 | controller |     - '          exit 1'
2026-03-14 09:56:26.796085 | controller |     - '        fi'
2026-03-14 09:56:26.796089 | controller |     - '        ]'
2026-03-14 09:56:26.796094 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.796098 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:56:26.796104 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN
2026-03-14 09:56:26.796109 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.796120 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.796125 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.796130 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.796134 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.796138 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.796143 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.796147 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.796151 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.796156 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.796160 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.796164 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.796169 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.796173 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.796177 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.796182 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.796186 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.796190 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.796195 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.796199 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.796203 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.796208 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.796212 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.796218 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:56:26.796223 | controller |       --decode'
2026-03-14 09:56:26.796230 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.796234 | controller |       get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:56:26.796239 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN
2026-03-14 09:56:26.796243 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.796247 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.796252 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.796256 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.796260 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.796270 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.796275 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.796279 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.796284 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.796288 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.796292 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.796297 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.796301 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.796305 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.796310 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.796314 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.796319 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.796323 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.796327 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.796332 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.796336 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.796349 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.796355 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.796362 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:56:26.796370 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.796375 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.796379 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.796384 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.796388 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.796392 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.796397 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.796401 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.796405 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.796409 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.796414 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.796425 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.796429 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.796434 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.796438 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.796443 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.796447 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.796451 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.796456 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.796460 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.796464 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.796469 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.796475 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.796479 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.796485 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.796489 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.796494 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.796498 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.796502 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.796507 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.796511 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.796515 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.796520 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.796524 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.796528 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.796533 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.796537 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.796541 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.796545 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.796550 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.796563 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.796568 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.796572 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.796576 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.796581 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:56:26.796585 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.796589 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:56:26.796605 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN
2026-03-14 09:56:26.796620 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.796624 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.796631 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.796635 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.796639 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.796644 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.796648 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.796652 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.796657 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.796661 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.796665 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.796670 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.796674 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.796678 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.796682 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.796687 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.796691 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.796695 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.796700 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.796704 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.796708 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.796713 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.796717 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.796721 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:56:26.796726 | controller |       --decode'
2026-03-14 09:56:26.796730 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.796741 | controller |       get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:56:26.796748 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN
2026-03-14 09:56:26.796752 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.796756 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.796766 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.796771 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.796775 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.796779 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.796784 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.796788 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.796792 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.796797 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.796801 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.796805 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.796810 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.796814 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.796818 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.796822 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.796827 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.796831 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.796835 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.796840 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.796846 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.796851 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.796884 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.796889 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:56:26.796894 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.796902 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.796906 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.796911 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.796915 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.796927 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.796931 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.796936 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.796940 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.796944 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.796949 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.796953 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.796957 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.796962 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.796966 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.796970 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.796980 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.796984 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.796989 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.796993 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.796997 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.797002 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.797006 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.797010 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.797021 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.797027 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.797032 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.797038 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.797042 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.797046 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.797051 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.797055 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.797059 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.797063 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.797074 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.797078 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.797083 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.797087 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.797091 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.797096 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.797100 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.797104 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.797109 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.797113 | controller |     - '    logger.go:42: 09:48:55 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.797117 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:56:26.797123 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:56:26.797128 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.797135 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.797139 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:56:26.797143 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.797148 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.797152 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.797156 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:56:26.797162 | controller |     - '          exit 1'
2026-03-14 09:56:26.797167 | controller |     - '        fi'
2026-03-14 09:56:26.797171 | controller |     - '        '
2026-03-14 09:56:26.797175 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.797180 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.797184 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.797188 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.797193 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.797197 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:56:26.797201 | controller |     - '          exit 1'
2026-03-14 09:56:26.797206 | controller |     - '        fi'
2026-03-14 09:56:26.797210 | controller |     - '        ]'
2026-03-14 09:56:26.797214 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.797219 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:56:26.797223 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN
2026-03-14 09:56:26.797227 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.797231 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.797236 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.797240 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.797244 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.797249 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.797253 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.797259 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.797263 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.797274 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.797278 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.797283 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.797287 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.797292 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.797296 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.797300 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.797304 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.797311 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.797315 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.797319 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.797324 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.797328 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.797332 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.797337 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:56:26.797341 | controller |       --decode'
2026-03-14 09:56:26.797345 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.797350 | controller |       get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:56:26.797354 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN
2026-03-14 09:56:26.797358 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.797363 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.797367 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.797371 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.797376 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.797380 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.797384 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.797389 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.797393 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.797399 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.797409 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.797413 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.797417 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.797431 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.797436 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.797442 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.797447 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.797451 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.797455 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.797460 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.797464 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.797468 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.797473 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.797477 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:56:26.797481 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.797486 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.797490 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.797494 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.797499 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.797503 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.797507 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.797512 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.797516 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.797520 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.797525 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.797531 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.797535 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.797540 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.797544 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.797548 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.797553 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.797559 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.797563 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.797573 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.797578 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.797582 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.797587 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.797591 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.797610 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.797615 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.797619 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.797623 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.797628 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.797632 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.797636 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.797641 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.797645 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.797649 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.797654 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.797660 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.797665 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.797669 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.797673 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.797678 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.797682 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.797688 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.797693 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.797697 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.797702 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:56:26.797708 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.797712 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:56:26.797717 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN
2026-03-14 09:56:26.797721 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.797725 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.797730 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.797740 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.797745 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.797750 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.797754 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.797758 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.797763 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.797767 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.797771 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.797776 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.797780 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.797784 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.797788 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.797795 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.797799 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.797804 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.797809 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.797814 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.797818 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.797823 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.797827 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.797831 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:56:26.797836 | controller |       --decode'
2026-03-14 09:56:26.797840 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.797844 | controller |       get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:56:26.797849 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN
2026-03-14 09:56:26.797853 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.797857 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.797862 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.797866 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.797870 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.797875 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.797879 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.797888 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.797893 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.797898 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.797902 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.797906 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.797911 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.797915 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.797919 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.797923 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.797928 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.797936 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.797940 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.797945 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.797949 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.797954 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.797958 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.797962 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:56:26.797967 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.797971 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.797976 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.797990 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.797995 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.798001 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.798016 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.798020 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.798025 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.798030 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.798035 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.798039 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.798044 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.798056 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.798064 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.798071 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.798076 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.798081 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.798092 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.798099 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.798104 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.798109 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.798117 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.798122 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.798127 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.798132 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.798140 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.798145 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.798149 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.798154 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.798158 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.798162 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.798167 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.798171 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.798175 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.798180 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.798184 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.798188 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.798193 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.798197 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.798201 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.798206 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.798217 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.798222 | controller |     - '    logger.go:42: 09:48:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.798226 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:56:26.798231 | controller |     - '    logger.go:42: 09:48:58 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:56:26.798235 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.798240 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.798244 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:56:26.798249 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.798253 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.798257 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.798262 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:56:26.798266 | controller |     - '          exit 1'
2026-03-14 09:56:26.798270 | controller |     - '        fi'
2026-03-14 09:56:26.798275 | controller |     - '        '
2026-03-14 09:56:26.798279 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.798283 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.798288 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.798292 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.798296 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.798301 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:56:26.798305 | controller |     - '          exit 1'
2026-03-14 09:56:26.798309 | controller |     - '        fi'
2026-03-14 09:56:26.798313 | controller |     - '        ]'
2026-03-14 09:56:26.798318 | controller |     - '    logger.go:42: 09:48:58 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.798322 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:56:26.798326 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN
2026-03-14 09:56:26.798331 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.798335 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.798341 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.798346 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.798350 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.798354 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.798359 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.798363 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.798367 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.798372 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.798378 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.798382 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.798386 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.798391 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.798395 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.798400 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.798404 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.798414 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.798418 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.798423 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.798427 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.798431 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.798436 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.798440 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:56:26.798444 | controller |       --decode'
2026-03-14 09:56:26.798450 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.798460 | controller |       get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:56:26.798465 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN
2026-03-14 09:56:26.798469 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.798480 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.798484 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.798488 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.798493 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.798499 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.798503 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.798507 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.798512 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.798518 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.798522 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.798527 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.798531 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.798535 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.798540 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.798544 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.798548 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.798553 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.798557 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.798561 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.798566 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.798576 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.798580 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.798585 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:56:26.798589 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.798605 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.798620 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.798625 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.798629 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.798634 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.798640 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.798645 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.798649 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.798653 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.798659 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.798664 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.798668 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.798672 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.798677 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.798681 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.798685 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.798690 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.798694 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.798698 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.798703 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.798707 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.798711 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.798716 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.798720 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.798724 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.798729 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.798733 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.798743 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.798748 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.798755 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.798759 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.798763 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.798767 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.798773 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.798778 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.798782 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.798786 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.798791 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.798795 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.798799 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.798804 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.798808 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.798991 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.798998 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:56:26.799002 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.799007 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:56:26.799011 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN
2026-03-14 09:56:26.799015 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.799020 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.799024 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.799028 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.799047 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.799054 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.799060 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.799064 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.799068 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.799072 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.799079 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.799090 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.799097 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.799101 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.799105 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.799109 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.799114 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.799118 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.799122 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.799126 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.799131 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.799135 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.799139 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.799143 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:56:26.799148 | controller |       --decode'
2026-03-14 09:56:26.799152 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.799156 | controller |       get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:56:26.799161 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN
2026-03-14 09:56:26.799165 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.799169 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.799173 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.799178 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.799182 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.799186 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.799191 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.799195 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.799199 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.799203 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.799208 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.799213 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.799220 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.799224 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.799233 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.799238 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.799242 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.799247 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.799251 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.799255 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.799259 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.799264 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.799268 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.799272 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:56:26.799276 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.799281 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.799285 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.799289 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.799293 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.799298 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.799302 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.799306 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.799310 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.799315 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.799319 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.799323 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.799329 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.799333 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.799340 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.799344 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.799348 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.799353 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.799357 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.799361 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.799365 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.799374 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.799379 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.799383 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.799388 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.799392 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.799396 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.799400 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.799405 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.799409 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.799413 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.799418 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.799422 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.799426 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.799430 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.799435 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.799441 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.799445 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.799452 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.799456 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.799460 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.799464 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.799469 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.799473 | controller |     - '    logger.go:42: 09:48:59 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.799477 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:56:26.799481 | controller |     - '    logger.go:42: 09:49:00 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:56:26.799485 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.799490 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.799494 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:56:26.799498 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.799502 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.799507 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.799511 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:56:26.799515 | controller |     - '          exit 1'
2026-03-14 09:56:26.799519 | controller |     - '        fi'
2026-03-14 09:56:26.799524 | controller |     - '        '
2026-03-14 09:56:26.799528 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.799532 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.799536 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.799545 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.799550 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.799554 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:56:26.799558 | controller |     - '          exit 1'
2026-03-14 09:56:26.799563 | controller |     - '        fi'
2026-03-14 09:56:26.799567 | controller |     - '        ]'
2026-03-14 09:56:26.799571 | controller |     - '    logger.go:42: 09:49:00 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.799575 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:56:26.799580 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN
2026-03-14 09:56:26.799584 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.799588 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.799592 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.799611 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.799618 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.799622 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.799626 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.799630 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.799635 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.799639 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.799643 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.799647 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.799652 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.799656 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.799660 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.799664 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.799669 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.799673 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.799679 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.799683 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.799687 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.799692 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.799696 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.799700 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:56:26.799705 | controller |       --decode'
2026-03-14 09:56:26.799709 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.799713 | controller |       get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:56:26.799717 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN
2026-03-14 09:56:26.799722 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.799732 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.799736 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.799742 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.799747 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.799751 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.799755 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.799759 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.799764 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.799768 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.799772 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.799776 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.799781 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.799785 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.799789 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.799794 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.799798 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.799802 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.799806 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.799811 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.799817 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.799821 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.799826 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.799830 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:56:26.799834 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.799839 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.799843 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.799847 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.799853 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.799857 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.799862 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.799871 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.799875 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.799879 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.799884 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.799888 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.799892 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.799896 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.799901 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.799905 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.799909 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.799913 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.799918 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.799922 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.799926 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.799930 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.799935 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.799939 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.799943 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.799949 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.799954 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.799958 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.799965 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.799970 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.799974 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.799978 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.799982 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.799987 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.799991 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.799995 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.800004 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.800009 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.800013 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.800017 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.800022 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.800026 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.800030 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.800034 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.800039 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:56:26.800043 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.800047 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:56:26.800058 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN
2026-03-14 09:56:26.800063 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.800067 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.800071 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.800075 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.800080 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.800093 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.800098 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.800102 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.800106 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.800110 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.800115 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.800119 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.800123 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.800127 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.800132 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.800136 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.800140 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.800144 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.800154 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.800158 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.800164 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.800168 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.800173 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.800177 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:56:26.800181 | controller |       --decode'
2026-03-14 09:56:26.800185 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.800190 | controller |       get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:56:26.800194 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN
2026-03-14 09:56:26.800198 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.800203 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.800207 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.800211 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.800217 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.800221 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.800226 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.800232 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.800236 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.800240 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.800244 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.800249 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.800253 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.800257 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.800261 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.800266 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.800270 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.800274 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.800278 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.800283 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.800287 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.800293 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.800297 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.800302 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:56:26.800306 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.800315 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.800320 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.800324 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.800328 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.800334 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.800339 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.800343 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.800347 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.800353 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.800358 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.800362 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.800366 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.800370 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.800374 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.800379 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.800383 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.800387 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.800391 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.800396 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.800400 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.800404 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.800408 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.800413 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.800417 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.800421 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.800425 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.800430 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.800434 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.800445 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.800450 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.800454 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.800458 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.800465 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.800469 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.800473 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.800477 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.800482 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.800486 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.800490 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.800494 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.800498 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.800503 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.800507 | controller |     - '    logger.go:42: 09:49:01 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.800511 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:56:26.800516 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:56:26.800520 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.800542 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.800546 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:56:26.800550 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:56:26.800555 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.800559 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.800563 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:56:26.800567 | controller |     - '          exit 1'
2026-03-14 09:56:26.800572 | controller |     - '        fi'
2026-03-14 09:56:26.800576 | controller |     - '        '
2026-03-14 09:56:26.800580 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:56:26.800584 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.800589 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:56:26.800593 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.800618 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.800624 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:56:26.800629 | controller |     - '          exit 1'
2026-03-14 09:56:26.800633 | controller |     - '        fi'
2026-03-14 09:56:26.800637 | controller |     - '        ]'
2026-03-14 09:56:26.800642 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.800646 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:56:26.800650 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN
2026-03-14 09:56:26.800654 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.800658 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.800663 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.800667 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.800677 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.800682 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.800686 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.800691 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.800695 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.800699 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.800703 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.800707 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.800712 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.800718 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.800722 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.800727 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.800731 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.800735 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.800740 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.800744 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.800748 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.800754 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.800759 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.800763 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:56:26.800767 | controller |       --decode'
2026-03-14 09:56:26.800772 | controller |     - '    logger.go:42: 09:49:02 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.800776 | controller |       get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:56:26.800780 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN
2026-03-14 09:56:26.800785 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.800789 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.800793 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.800797 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.800802 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.800806 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.800810 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.800815 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.800819 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.800829 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.800834 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.800838 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.800842 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.800847 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.800851 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.800857 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.800862 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.800866 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.800870 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.800877 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.800882 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.800886 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.800890 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.800895 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:56:26.800899 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.800903 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.800908 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.800912 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.800916 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.800921 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.800925 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.800929 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.800934 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.800938 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.800942 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.800946 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.800951 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.800955 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.800959 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.800969 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.800974 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.800980 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.800984 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.800989 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.800995 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.800999 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.801004 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.801008 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.801012 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.801016 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.801021 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.801025 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.801029 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.801034 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.801038 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.801042 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.801047 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.801051 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.801055 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.801060 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.801064 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.801068 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.801073 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.801077 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.801081 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.801087 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.801092 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.801096 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.801100 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:56:26.801110 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.801116 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:56:26.801121 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN
2026-03-14 09:56:26.801125 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.801130 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.801134 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.801138 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.801143 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.801147 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.801151 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.801155 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.801160 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.801164 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.801168 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.801173 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.801177 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.801181 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.801186 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.801190 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.801194 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.801198 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.801203 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.801207 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.801211 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.801216 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.801220 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.801226 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:56:26.801231 | controller |       --decode'
2026-03-14 09:56:26.801235 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:56:26.801239 | controller |       get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:56:26.801244 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN
2026-03-14 09:56:26.801248 | controller |       CERTIFICATE-----'
2026-03-14 09:56:26.801252 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.801265 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.801270 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.801274 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.801279 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.801283 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.801287 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.801292 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.801296 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.801300 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.801305 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.801309 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.801315 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.801319 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.801323 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.801328 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.801332 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.801336 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.801340 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.801345 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.801351 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.801355 | controller |       CERTIFICATE-----'''
2026-03-14 09:56:26.801359 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:56:26.801364 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.801368 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.801372 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.801377 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.801381 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.801385 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.801389 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.801394 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.801403 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.801407 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.801412 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.801416 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.801420 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.801425 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.801429 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.801433 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.801437 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.801441 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.801446 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.801450 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.801454 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.801458 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.801467 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.801471 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtjCCAh6gAwIBAgIQQDVTlXzFRHe8rKlQXUgBRzANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:56:26.801475 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDk0ODM3WhcNMzEw'
2026-03-14 09:56:26.801479 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | MzEzMDk0ODM3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtepU'
2026-03-14 09:56:26.801484 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | htHRZpecqtbLjd59iCMVwQSL6yjmbI00Ds3qIuz8Dwf/aCLlMiL28HYMBMB/Xosn'
2026-03-14 09:56:26.801488 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | F9OCvHU2Fs+FCGT0uyT4BWQiJKcGiTLP6fExF6WU2jrRUzUrrccR2pInb5SCrrYb'
2026-03-14 09:56:26.801492 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | UU79MDvfyjKCv669gadcdoc7EcoUUtwwR7+8QeIj23XBBD9W5bnYzu+k8YR0cSve'
2026-03-14 09:56:26.801496 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | nGribTCHrQTrfpYvK3s9JFh5kpk0BtyzU0mWkA7AtyBOx3SsXBAlK1/kIAv7kNn2'
2026-03-14 09:56:26.801501 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | CKA+ybu05qPuq/duku6EYEdQU59zzcjQQC7XtHkzp/YoBDO0w3OcKu40o8WEH47f'
2026-03-14 09:56:26.801505 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | fwNBsYE7U7pQs7WmGQIDAQABo4GRMIGOMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:56:26.801509 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFMoZkNaTOHGD'
2026-03-14 09:56:26.801513 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | lLYu0oITkRA7ataaMDgGA1UdEQEB/wQuMCyCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:56:26.801517 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Y2hlci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAqaM71u6t'
2026-03-14 09:56:26.801522 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 7UmoUcIyoenmm3c1U5QWAotTZqwYRmfEXg0snVvyPEy3HAppECAAbTb6dyAMPMDa'
2026-03-14 09:56:26.801534 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | Iq4n+xvYE/3IoK9guL5WGRWI8yWfqCM34TAJA2KuU/8cTY61YYGXqYSeGCLDNTkx'
2026-03-14 09:56:26.801543 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | i0ZD+LzOr6xbs59ia6UTe5OMjJNXPnG+FWuE5NDn0LaMco4oDbpUMHquHRRpZSCo'
2026-03-14 09:56:26.801548 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | JbOlL1GSmbjNo7Qwltxq9Xn+ZvQDyVogVsB1LjInTGoJ7AGjGgLunMtWiCp2SQrj'
2026-03-14 09:56:26.801552 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | 682okPU+SgjY9Md5DooExJK/yA7Ihf664eGVkTQmrWrlA3kq9Syf/unaQIycLYok'
2026-03-14 09:56:26.801556 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | SLm3Bkg3rbfEUn6ym9Hp8oEpxVz7e3MTeI/rrGhzLtQQ6JmKGhkQy6RkePIycN7g'
2026-03-14 09:56:26.801561 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | +r6MMPFWhSnJ2SW7xsbCJS2waIWzQ5t6ddE+k0LfC8P5LB6waXuaVc4WWVZHGkX+'
2026-03-14 09:56:26.801565 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | u5vEJh6UGx5zn6ov0NSFDHgg8hI0iVLpWAM2o/WSEaBGQyu/L/9xz2Jy'
2026-03-14 09:56:26.801569 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:56:26.801575 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:56:26.801580 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/1-deploy-with-tlse | test
2026-03-14 09:56:26.801584 | controller |       step completed 1-deploy-with-tlse'
2026-03-14 09:56:26.801588 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801605 | controller |       | starting test step 2-change-public-svc-certificate'
2026-03-14 09:56:26.801611 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801616 | controller |       | running command: [sh -c oc patch Certificate -n $NAMESPACE watcher-public-svc
2026-03-14 09:56:26.801620 | controller |       --type=''json'' -p=''[{"op": "replace", "path": "/spec/dnsNames", "value":[''watcher-public.watcher-kuttl-default.svc'',
2026-03-14 09:56:26.801624 | controller |       ''watcher-public.watcher-kuttl-default.svc.cluster.local'']}]'''
2026-03-14 09:56:26.801628 | controller |     - '        ]'
2026-03-14 09:56:26.801632 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801637 | controller |       | certificate.cert-manager.io/watcher-public-svc patched'
2026-03-14 09:56:26.801643 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801647 | controller |       | running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.801651 | controller |     - '        svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat
2026-03-14 09:56:26.801656 | controller |       /etc/pki/tls/certs/public.crt)'
2026-03-14 09:56:26.801660 | controller |     - '        secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}''
2026-03-14 09:56:26.801664 | controller |       | base64 --decode)'
2026-03-14 09:56:26.801668 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.801673 | controller |     - '        if [ "${svc_cert}" != "${secret_cert}" ]; then'
2026-03-14 09:56:26.801677 | controller |     - '          exit 1'
2026-03-14 09:56:26.801681 | controller |     - '        fi'
2026-03-14 09:56:26.801685 | controller |     - '        ]'
2026-03-14 09:56:26.801690 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801694 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:56:26.801698 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801702 | controller |       | + svc_cert=''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.801707 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801711 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.801715 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801719 | controller |       | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.801724 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801728 | controller |       | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.801732 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801736 | controller |       | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.801740 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801745 | controller |       | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.801749 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801753 | controller |       | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.801757 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801762 | controller |       | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.801766 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801770 | controller |       | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.801780 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801788 | controller |       | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.801792 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801797 | controller |       | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.801801 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801805 | controller |       | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.801809 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801813 | controller |       | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.801818 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801822 | controller |       | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.801826 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801830 | controller |       | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.801834 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801838 | controller |       | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.801843 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801847 | controller |       | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.801851 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801855 | controller |       | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.801859 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801864 | controller |       | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.801868 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801872 | controller |       | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.801876 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801881 | controller |       | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.801885 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801889 | controller |       | -----END CERTIFICATE-----'''
2026-03-14 09:56:26.801893 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801897 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:56:26.801902 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801906 | controller |       | ++ base64 --decode'
2026-03-14 09:56:26.801910 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801914 | controller |       | + secret_cert=''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.801919 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801923 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.801927 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801931 | controller |       | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.801936 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801941 | controller |       | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.801949 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801953 | controller |       | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.801957 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801962 | controller |       | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.801966 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801970 | controller |       | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.801974 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801978 | controller |       | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.801983 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801987 | controller |       | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.801991 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.801995 | controller |       | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.802006 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802010 | controller |       | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.802014 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802019 | controller |       | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.802023 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802027 | controller |       | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.802032 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802036 | controller |       | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.802040 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802044 | controller |       | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.802048 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802052 | controller |       | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.802057 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802061 | controller |       | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.802065 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802069 | controller |       | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.802073 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802078 | controller |       | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.802082 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802086 | controller |       | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.802090 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802094 | controller |       | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.802099 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802103 | controller |       | -----END CERTIFICATE-----'''
2026-03-14 09:56:26.802107 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802113 | controller |       | + ''['' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.802118 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802122 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.802126 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802130 | controller |       | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.802135 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802139 | controller |       | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.802143 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802147 | controller |       | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.802152 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802156 | controller |       | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.802160 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802164 | controller |       | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.802168 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802173 | controller |       | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.802177 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802181 | controller |       | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.802185 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802189 | controller |       | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.802194 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802198 | controller |       | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.802202 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802206 | controller |       | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.802210 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802215 | controller |       | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.802226 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802231 | controller |       | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.802235 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802239 | controller |       | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.802244 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802248 | controller |       | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.802252 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802256 | controller |       | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.802261 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802265 | controller |       | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.802269 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802275 | controller |       | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.802279 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802283 | controller |       | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.802287 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802292 | controller |       | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.802296 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802302 | controller |       | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.802306 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802310 | controller |       | MIIDszCCAhugAwIBAgIRAMhUiOy62VoKQLz1SRYTJ9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.802314 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802318 | controller |       | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTQ4MzhaFw0zMTAz'
2026-03-14 09:56:26.802323 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802327 | controller |       | MTMwOTQ4MzhaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPMnhL'
2026-03-14 09:56:26.802331 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802335 | controller |       | FyT6i+RM5IV9XjRLfeS+Md8Zr8wVYBGxBoc4gz7Ok56Mq5ZrQCEJKR9wGXm90xuM'
2026-03-14 09:56:26.802340 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802344 | controller |       | UDX7FH+Z9vGYJ8BtJG/nSyRsb48rgvU99ZJB39M7lT4YtUdwOtt0yUGGAloiSpBp'
2026-03-14 09:56:26.802348 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802352 | controller |       | vows0Q6iKifU57kyzTj7vyfFsNpuCXpaaRPZj2k4tfEdUCSrLWiyU9VYYLKdfhJs'
2026-03-14 09:56:26.802357 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802361 | controller |       | vMSOmi706bkctr+3VoP8S9QRxA8/HIs2eoEQfh0/OjEOsteZOEk7tv6+A1D6zpg/'
2026-03-14 09:56:26.802365 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802369 | controller |       | 6/c+KG36AE2JWqJUyCnuX9FH532f9OLqS7Apoani5Fb7AGFSXSDyBc8SKWMlqrRR'
2026-03-14 09:56:26.802373 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802378 | controller |       | Zw/WLGuDCChKJHWXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:56:26.802382 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802386 | controller |       | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUBwWTbQqaeYvX'
2026-03-14 09:56:26.802390 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802394 | controller |       | 7NMhyiOPWCqMQRowNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:56:26.802399 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802403 | controller |       | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAEmCnND/AzwOh'
2026-03-14 09:56:26.802407 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802411 | controller |       | gp7eDfgCQt2rlRgRwA7I/cOytAfNMnl+BliNHapoan+Ty86JgruSw/wvA74K6XZg'
2026-03-14 09:56:26.802416 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802420 | controller |       | V2KgmnnLbo2OLmfYIl0d7AoERw/kIfMbXJY/CMHjAQ1SQyUlgHyIhb7YyMry4zxO'
2026-03-14 09:56:26.802424 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802428 | controller |       | Ix9XljgixFBCqsm7HzPs38XBMyWQTIgyNJI3bmfTzCUIJRGEQCmWAWu5cg+x5CrK'
2026-03-14 09:56:26.802434 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802445 | controller |       | wDjHIDQQb2YSsqYlFq32j1pZRgRt2jZNmF90KIetw8jlyITBk9Rb2oHUvSkqB/1f'
2026-03-14 09:56:26.802449 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802453 | controller |       | FsYMgRfZKCEvuLcTRGvnVok0E0RxFXtcElcIGIUekAlh8JCXc9QkXqTChCHtMKOC'
2026-03-14 09:56:26.802458 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802462 | controller |       | /2Ha1I/RvfINjIOTOHwkYs2SGtcvdofGk58o9IYuo2FmCJE5nXaQS9EIy4Yf5s6T'
2026-03-14 09:56:26.802466 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802470 | controller |       | 5JuPtk8b9wnE/Kouc+34Y7U/uOS7omS3ylRPt+YVtZTwGScLqDC/a/8eomEimooA'
2026-03-14 09:56:26.802474 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802479 | controller |       | QvjquXCitigQkLNJBUfagbm3F49byKBcGFc2YvL5bfKitmd6W4Hm'
2026-03-14 09:56:26.802483 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802487 | controller |       | -----END CERTIFICATE-----'' '']'''
2026-03-14 09:56:26.802491 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:56:26.802496 | controller |       | test step completed 2-change-public-svc-certificate'
2026-03-14 09:56:26.802502 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802506 | controller |       | starting test step 3-change-internal-svc-certificate'
2026-03-14 09:56:26.802510 | controller |     - '    logger.go:42: 09:49:03 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802514 | controller |       | running command: [sh -c oc patch Certificate -n $NAMESPACE watcher-internal-svc
2026-03-14 09:56:26.802518 | controller |       --type=''json'' -p=''[{"op": "replace", "path": "/spec/dnsNames", "value":[''watcher-internal.watcher-kuttl-default.svc'',
2026-03-14 09:56:26.802523 | controller |       ''watcher-internal.watcher-kuttl-default.svc.cluster.local'']}]'''
2026-03-14 09:56:26.802527 | controller |     - '        ]'
2026-03-14 09:56:26.802531 | controller |     - '    logger.go:42: 09:49:04 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802536 | controller |       | certificate.cert-manager.io/watcher-internal-svc patched'
2026-03-14 09:56:26.802540 | controller |     - '    logger.go:42: 09:49:04 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802544 | controller |       | running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.802548 | controller |     - '        svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat
2026-03-14 09:56:26.802552 | controller |       /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.802557 | controller |     - '        secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o
2026-03-14 09:56:26.802561 | controller |       jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.802565 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.802569 | controller |     - '        if [ "${svc_cert}" != "${secret_cert}" ]; then'
2026-03-14 09:56:26.802573 | controller |     - '          exit 1'
2026-03-14 09:56:26.802578 | controller |     - '        fi'
2026-03-14 09:56:26.802582 | controller |     - '        ]'
2026-03-14 09:56:26.802586 | controller |     - '    logger.go:42: 09:49:04 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802591 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:56:26.802605 | controller |     - '    logger.go:42: 09:49:05 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802611 | controller |       | error: Internal error occurred: error executing command in container: container
2026-03-14 09:56:26.802615 | controller |       is not created or running'
2026-03-14 09:56:26.802620 | controller |     - '    logger.go:42: 09:49:05 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802626 | controller |       | + svc_cert='
2026-03-14 09:56:26.802630 | controller |     - '    logger.go:42: 09:49:06 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802634 | controller |       | running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.802639 | controller |     - '        svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat
2026-03-14 09:56:26.802643 | controller |       /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.802647 | controller |     - '        secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o
2026-03-14 09:56:26.802651 | controller |       jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.802655 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.802660 | controller |     - '        if [ "${svc_cert}" != "${secret_cert}" ]; then'
2026-03-14 09:56:26.802664 | controller |     - '          exit 1'
2026-03-14 09:56:26.802668 | controller |     - '        fi'
2026-03-14 09:56:26.802672 | controller |     - '        ]'
2026-03-14 09:56:26.802677 | controller |     - '    logger.go:42: 09:49:06 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802681 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:56:26.802685 | controller |     - '    logger.go:42: 09:49:06 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802689 | controller |       | error: cannot exec into a container in a completed pod; current phase is Failed'
2026-03-14 09:56:26.802694 | controller |     - '    logger.go:42: 09:49:06 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802698 | controller |       | + svc_cert='
2026-03-14 09:56:26.802702 | controller |     - '    logger.go:42: 09:49:07 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802706 | controller |       | running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.802717 | controller |     - '        svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat
2026-03-14 09:56:26.802722 | controller |       /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.802726 | controller |     - '        secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o
2026-03-14 09:56:26.802730 | controller |       jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.802734 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.802739 | controller |     - '        if [ "${svc_cert}" != "${secret_cert}" ]; then'
2026-03-14 09:56:26.802743 | controller |     - '          exit 1'
2026-03-14 09:56:26.802747 | controller |     - '        fi'
2026-03-14 09:56:26.802751 | controller |     - '        ]'
2026-03-14 09:56:26.802756 | controller |     - '    logger.go:42: 09:49:07 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802760 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:56:26.802764 | controller |     - '    logger.go:42: 09:49:07 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802768 | controller |       | error: unable to upgrade connection: container not found ("watcher-api")'
2026-03-14 09:56:26.802773 | controller |     - '    logger.go:42: 09:49:07 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802777 | controller |       | + svc_cert='
2026-03-14 09:56:26.802781 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802785 | controller |       | running command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.802789 | controller |     - '        svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat
2026-03-14 09:56:26.802794 | controller |       /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:56:26.802798 | controller |     - '        secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o
2026-03-14 09:56:26.802802 | controller |       jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:56:26.802806 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:56:26.802811 | controller |     - '        if [ "${svc_cert}" != "${secret_cert}" ]; then'
2026-03-14 09:56:26.802817 | controller |     - '          exit 1'
2026-03-14 09:56:26.802821 | controller |     - '        fi'
2026-03-14 09:56:26.802825 | controller |     - '        ]'
2026-03-14 09:56:26.802829 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802834 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:56:26.802838 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802842 | controller |       | + svc_cert=''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.802846 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802851 | controller |       | MIID8TCCAlmgAwIBAgIRANEJGjnrkU5zntzPj7vLEQUwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.802857 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802861 | controller |       | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5NDkwNFoXDTMx'
2026-03-14 09:56:26.802865 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802870 | controller |       | MDMxMzA5NDkwNFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK2c'
2026-03-14 09:56:26.802874 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802878 | controller |       | pgzS28NUzYI+nLrg8YANmwvnjV799rH8Mt2MKso4tswc2tn/+L4eZmTkxibhbs2M'
2026-03-14 09:56:26.802882 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802886 | controller |       | TwKhPdGOsTdO+5uf3eq5uYBGUJJQzLg6WUjlaIE4/y8ELdXwn9Si8JNGg8TUO2+o'
2026-03-14 09:56:26.802891 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802895 | controller |       | dJJous/W/rG11NUGJeaOJQkEuTNvmVEq3wevESe9WJhTBdxOmtfR1V6qHbL9Bbi6'
2026-03-14 09:56:26.802899 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802903 | controller |       | djRSEFWTYTojAtbB+9ZluXXXr986tLaYQi7hrc8VhNp9wAYxnmWAV6h6OtQsv4oR'
2026-03-14 09:56:26.802908 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802912 | controller |       | L5nRqzNvLEnrIxSN3XM2kQQL/yN7rh1fb3j8bNUtB0opUjm0SSFsP0/k0fmE4WMu'
2026-03-14 09:56:26.802916 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802920 | controller |       | A5Sg1np9ujvpUTwAq58CAwEAAaOByzCByDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:56:26.802924 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802929 | controller |       | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBTKGZDWkzhx'
2026-03-14 09:56:26.802933 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802939 | controller |       | g5S2LtKCE5EQO2rWmjByBgNVHREBAf8EaDBmgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:56:26.802943 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802947 | controller |       | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmOCOHdhdGNoZXItaW50ZXJuYWwud2F0Y2hl'
2026-03-14 09:56:26.802951 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802963 | controller |       | ci1rdXR0bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUA'
2026-03-14 09:56:26.802968 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802972 | controller |       | A4IBgQDNKBRriKybBx92y2bamYy1IsYsPZ9IQSIkLavcYm+ytlMKKv1bjgJ13AWo'
2026-03-14 09:56:26.802976 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802980 | controller |       | fts0EPcJfblCY6i6hkFYh6GXuAlrW5+HiEsGcybOtH/tYVksh3qzDUKmKgJsPvAA'
2026-03-14 09:56:26.802984 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.802992 | controller |       | lzWsTfnTzgvmQK+Cfh6dg7PwOgkoB8Lv+vnYzbX9gL9Hw6qYObv/dgoGJTuj4tiG'
2026-03-14 09:56:26.802996 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803000 | controller |       | kn4mVPYVCdpAQ5jQW9iGEuiLYvQ6/klSOD4YF2JqF3wf759g9yBOacFSIbd4xn9I'
2026-03-14 09:56:26.803005 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803009 | controller |       | uMwr9eYjZJX1N2Yg8nYtBpi9WupGIWfBVf+7BygXxozT1KG+D6+bWS4J2Ngx6jbo'
2026-03-14 09:56:26.803013 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803017 | controller |       | SN+rmc+mwymN7+0GLTYxXH9vEJgO1648TGlSn5S4f+ONz8av68Qwo1RsYHLIHI9h'
2026-03-14 09:56:26.803021 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803026 | controller |       | xhsX0pMrarGyXfryX9YLnnxxpYQ4V90j8AxPawoFpj+8gNXa6X174+dKfA2QuNUQ'
2026-03-14 09:56:26.803030 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803035 | controller |       | h6yFKf0wvwih9nnYS8SsyxhZTDZP70pYQqdziTjXNUsctIn+WGrZcsZXoMqTF179'
2026-03-14 09:56:26.803056 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803060 | controller |       | +SCIwhM='
2026-03-14 09:56:26.803064 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803068 | controller |       | -----END CERTIFICATE-----'''
2026-03-14 09:56:26.803073 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803077 | controller |       | ++ base64 --decode'
2026-03-14 09:56:26.803081 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803085 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:56:26.803090 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803094 | controller |       | + secret_cert=''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.803098 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803103 | controller |       | MIID8TCCAlmgAwIBAgIRANEJGjnrkU5zntzPj7vLEQUwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.803107 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803111 | controller |       | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5NDkwNFoXDTMx'
2026-03-14 09:56:26.803115 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803119 | controller |       | MDMxMzA5NDkwNFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK2c'
2026-03-14 09:56:26.803124 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803128 | controller |       | pgzS28NUzYI+nLrg8YANmwvnjV799rH8Mt2MKso4tswc2tn/+L4eZmTkxibhbs2M'
2026-03-14 09:56:26.803132 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803136 | controller |       | TwKhPdGOsTdO+5uf3eq5uYBGUJJQzLg6WUjlaIE4/y8ELdXwn9Si8JNGg8TUO2+o'
2026-03-14 09:56:26.803141 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803145 | controller |       | dJJous/W/rG11NUGJeaOJQkEuTNvmVEq3wevESe9WJhTBdxOmtfR1V6qHbL9Bbi6'
2026-03-14 09:56:26.803151 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803155 | controller |       | djRSEFWTYTojAtbB+9ZluXXXr986tLaYQi7hrc8VhNp9wAYxnmWAV6h6OtQsv4oR'
2026-03-14 09:56:26.803159 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803164 | controller |       | L5nRqzNvLEnrIxSN3XM2kQQL/yN7rh1fb3j8bNUtB0opUjm0SSFsP0/k0fmE4WMu'
2026-03-14 09:56:26.803172 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803176 | controller |       | A5Sg1np9ujvpUTwAq58CAwEAAaOByzCByDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:56:26.803180 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803185 | controller |       | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBTKGZDWkzhx'
2026-03-14 09:56:26.803189 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803193 | controller |       | g5S2LtKCE5EQO2rWmjByBgNVHREBAf8EaDBmgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:56:26.803197 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803209 | controller |       | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmOCOHdhdGNoZXItaW50ZXJuYWwud2F0Y2hl'
2026-03-14 09:56:26.803213 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803217 | controller |       | ci1rdXR0bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUA'
2026-03-14 09:56:26.803222 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803226 | controller |       | A4IBgQDNKBRriKybBx92y2bamYy1IsYsPZ9IQSIkLavcYm+ytlMKKv1bjgJ13AWo'
2026-03-14 09:56:26.803230 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803234 | controller |       | fts0EPcJfblCY6i6hkFYh6GXuAlrW5+HiEsGcybOtH/tYVksh3qzDUKmKgJsPvAA'
2026-03-14 09:56:26.803239 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803243 | controller |       | lzWsTfnTzgvmQK+Cfh6dg7PwOgkoB8Lv+vnYzbX9gL9Hw6qYObv/dgoGJTuj4tiG'
2026-03-14 09:56:26.803247 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803251 | controller |       | kn4mVPYVCdpAQ5jQW9iGEuiLYvQ6/klSOD4YF2JqF3wf759g9yBOacFSIbd4xn9I'
2026-03-14 09:56:26.803257 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803261 | controller |       | uMwr9eYjZJX1N2Yg8nYtBpi9WupGIWfBVf+7BygXxozT1KG+D6+bWS4J2Ngx6jbo'
2026-03-14 09:56:26.803265 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803269 | controller |       | SN+rmc+mwymN7+0GLTYxXH9vEJgO1648TGlSn5S4f+ONz8av68Qwo1RsYHLIHI9h'
2026-03-14 09:56:26.803273 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803278 | controller |       | xhsX0pMrarGyXfryX9YLnnxxpYQ4V90j8AxPawoFpj+8gNXa6X174+dKfA2QuNUQ'
2026-03-14 09:56:26.803282 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803286 | controller |       | h6yFKf0wvwih9nnYS8SsyxhZTDZP70pYQqdziTjXNUsctIn+WGrZcsZXoMqTF179'
2026-03-14 09:56:26.803290 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803295 | controller |       | +SCIwhM='
2026-03-14 09:56:26.803301 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803305 | controller |       | -----END CERTIFICATE-----'''
2026-03-14 09:56:26.803310 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803314 | controller |       | + ''['' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.803318 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803322 | controller |       | MIID8TCCAlmgAwIBAgIRANEJGjnrkU5zntzPj7vLEQUwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.803327 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803331 | controller |       | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5NDkwNFoXDTMx'
2026-03-14 09:56:26.803335 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803341 | controller |       | MDMxMzA5NDkwNFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK2c'
2026-03-14 09:56:26.803345 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803349 | controller |       | pgzS28NUzYI+nLrg8YANmwvnjV799rH8Mt2MKso4tswc2tn/+L4eZmTkxibhbs2M'
2026-03-14 09:56:26.803354 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803358 | controller |       | TwKhPdGOsTdO+5uf3eq5uYBGUJJQzLg6WUjlaIE4/y8ELdXwn9Si8JNGg8TUO2+o'
2026-03-14 09:56:26.803364 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803368 | controller |       | dJJous/W/rG11NUGJeaOJQkEuTNvmVEq3wevESe9WJhTBdxOmtfR1V6qHbL9Bbi6'
2026-03-14 09:56:26.803372 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803376 | controller |       | djRSEFWTYTojAtbB+9ZluXXXr986tLaYQi7hrc8VhNp9wAYxnmWAV6h6OtQsv4oR'
2026-03-14 09:56:26.803381 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803385 | controller |       | L5nRqzNvLEnrIxSN3XM2kQQL/yN7rh1fb3j8bNUtB0opUjm0SSFsP0/k0fmE4WMu'
2026-03-14 09:56:26.803389 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803393 | controller |       | A5Sg1np9ujvpUTwAq58CAwEAAaOByzCByDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:56:26.803398 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803402 | controller |       | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBTKGZDWkzhx'
2026-03-14 09:56:26.803406 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803410 | controller |       | g5S2LtKCE5EQO2rWmjByBgNVHREBAf8EaDBmgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:56:26.803415 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803419 | controller |       | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmOCOHdhdGNoZXItaW50ZXJuYWwud2F0Y2hl'
2026-03-14 09:56:26.803423 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803433 | controller |       | ci1rdXR0bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUA'
2026-03-14 09:56:26.803438 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803444 | controller |       | A4IBgQDNKBRriKybBx92y2bamYy1IsYsPZ9IQSIkLavcYm+ytlMKKv1bjgJ13AWo'
2026-03-14 09:56:26.803448 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803452 | controller |       | fts0EPcJfblCY6i6hkFYh6GXuAlrW5+HiEsGcybOtH/tYVksh3qzDUKmKgJsPvAA'
2026-03-14 09:56:26.803456 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803461 | controller |       | lzWsTfnTzgvmQK+Cfh6dg7PwOgkoB8Lv+vnYzbX9gL9Hw6qYObv/dgoGJTuj4tiG'
2026-03-14 09:56:26.803465 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803469 | controller |       | kn4mVPYVCdpAQ5jQW9iGEuiLYvQ6/klSOD4YF2JqF3wf759g9yBOacFSIbd4xn9I'
2026-03-14 09:56:26.803473 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803478 | controller |       | uMwr9eYjZJX1N2Yg8nYtBpi9WupGIWfBVf+7BygXxozT1KG+D6+bWS4J2Ngx6jbo'
2026-03-14 09:56:26.803482 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803486 | controller |       | SN+rmc+mwymN7+0GLTYxXH9vEJgO1648TGlSn5S4f+ONz8av68Qwo1RsYHLIHI9h'
2026-03-14 09:56:26.803490 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803495 | controller |       | xhsX0pMrarGyXfryX9YLnnxxpYQ4V90j8AxPawoFpj+8gNXa6X174+dKfA2QuNUQ'
2026-03-14 09:56:26.803499 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803505 | controller |       | h6yFKf0wvwih9nnYS8SsyxhZTDZP70pYQqdziTjXNUsctIn+WGrZcsZXoMqTF179'
2026-03-14 09:56:26.803509 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803513 | controller |       | +SCIwhM='
2026-03-14 09:56:26.803518 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803522 | controller |       | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:56:26.803526 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803530 | controller |       | MIID8TCCAlmgAwIBAgIRANEJGjnrkU5zntzPj7vLEQUwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:56:26.803535 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803539 | controller |       | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5NDkwNFoXDTMx'
2026-03-14 09:56:26.803543 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803547 | controller |       | MDMxMzA5NDkwNFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK2c'
2026-03-14 09:56:26.803551 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803556 | controller |       | pgzS28NUzYI+nLrg8YANmwvnjV799rH8Mt2MKso4tswc2tn/+L4eZmTkxibhbs2M'
2026-03-14 09:56:26.803560 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803564 | controller |       | TwKhPdGOsTdO+5uf3eq5uYBGUJJQzLg6WUjlaIE4/y8ELdXwn9Si8JNGg8TUO2+o'
2026-03-14 09:56:26.803568 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803572 | controller |       | dJJous/W/rG11NUGJeaOJQkEuTNvmVEq3wevESe9WJhTBdxOmtfR1V6qHbL9Bbi6'
2026-03-14 09:56:26.803577 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803581 | controller |       | djRSEFWTYTojAtbB+9ZluXXXr986tLaYQi7hrc8VhNp9wAYxnmWAV6h6OtQsv4oR'
2026-03-14 09:56:26.803585 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803589 | controller |       | L5nRqzNvLEnrIxSN3XM2kQQL/yN7rh1fb3j8bNUtB0opUjm0SSFsP0/k0fmE4WMu'
2026-03-14 09:56:26.803603 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803610 | controller |       | A5Sg1np9ujvpUTwAq58CAwEAAaOByzCByDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:56:26.803614 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803618 | controller |       | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBTKGZDWkzhx'
2026-03-14 09:56:26.803622 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803627 | controller |       | g5S2LtKCE5EQO2rWmjByBgNVHREBAf8EaDBmgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:56:26.803631 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803635 | controller |       | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmOCOHdhdGNoZXItaW50ZXJuYWwud2F0Y2hl'
2026-03-14 09:56:26.803639 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803644 | controller |       | ci1rdXR0bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUA'
2026-03-14 09:56:26.803648 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803652 | controller |       | A4IBgQDNKBRriKybBx92y2bamYy1IsYsPZ9IQSIkLavcYm+ytlMKKv1bjgJ13AWo'
2026-03-14 09:56:26.803663 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803667 | controller |       | fts0EPcJfblCY6i6hkFYh6GXuAlrW5+HiEsGcybOtH/tYVksh3qzDUKmKgJsPvAA'
2026-03-14 09:56:26.803674 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803678 | controller |       | lzWsTfnTzgvmQK+Cfh6dg7PwOgkoB8Lv+vnYzbX9gL9Hw6qYObv/dgoGJTuj4tiG'
2026-03-14 09:56:26.803684 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803688 | controller |       | kn4mVPYVCdpAQ5jQW9iGEuiLYvQ6/klSOD4YF2JqF3wf759g9yBOacFSIbd4xn9I'
2026-03-14 09:56:26.803692 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803697 | controller |       | uMwr9eYjZJX1N2Yg8nYtBpi9WupGIWfBVf+7BygXxozT1KG+D6+bWS4J2Ngx6jbo'
2026-03-14 09:56:26.803701 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803707 | controller |       | SN+rmc+mwymN7+0GLTYxXH9vEJgO1648TGlSn5S4f+ONz8av68Qwo1RsYHLIHI9h'
2026-03-14 09:56:26.803711 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803715 | controller |       | xhsX0pMrarGyXfryX9YLnnxxpYQ4V90j8AxPawoFpj+8gNXa6X174+dKfA2QuNUQ'
2026-03-14 09:56:26.803719 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803723 | controller |       | h6yFKf0wvwih9nnYS8SsyxhZTDZP70pYQqdziTjXNUsctIn+WGrZcsZXoMqTF179'
2026-03-14 09:56:26.803729 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803733 | controller |       | +SCIwhM='
2026-03-14 09:56:26.803738 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803742 | controller |       | -----END CERTIFICATE-----'' '']'''
2026-03-14 09:56:26.803746 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:56:26.803750 | controller |       | test step completed 3-change-internal-svc-certificate'
2026-03-14 09:56:26.803754 | controller |     - '    logger.go:42: 09:49:08 | watcher-tls-certs-change/4-cleanup-watcher | starting
2026-03-14 09:56:26.803759 | controller |       test step 4-cleanup-watcher'
2026-03-14 09:56:26.803763 | controller |     - '    logger.go:42: 09:49:19 | watcher-tls-certs-change/4-cleanup-watcher | test
2026-03-14 09:56:26.803767 | controller |       step completed 4-cleanup-watcher'
2026-03-14 09:56:26.803771 | controller |     - '    logger.go:42: 09:49:19 | watcher-tls-certs-change/5-clenaup-certs | starting
2026-03-14 09:56:26.803776 | controller |       test step 5-clenaup-certs'
2026-03-14 09:56:26.803780 | controller |     - '    logger.go:42: 09:49:19 | watcher-tls-certs-change/5-clenaup-certs | test step
2026-03-14 09:56:26.803784 | controller |       completed 5-clenaup-certs'
2026-03-14 09:56:26.803788 | controller |     - '    logger.go:42: 09:49:19 | watcher-tls-certs-change | skipping kubernetes event
2026-03-14 09:56:26.803793 | controller |       logging'
2026-03-14 09:56:26.803799 | controller |     - === CONT  kuttl/harness/watcher-tls
2026-03-14 09:56:26.803804 | controller |     - '    logger.go:42: 09:49:19 | watcher-tls | Skipping creation of user-supplied namespace:
2026-03-14 09:56:26.803808 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.803812 | controller |     - '    logger.go:42: 09:49:19 | watcher-tls/0-cleanup-watcher | starting test step
2026-03-14 09:56:26.803816 | controller |       0-cleanup-watcher'
2026-03-14 09:56:26.803821 | controller |     - '    logger.go:42: 09:49:19 | watcher-tls/0-cleanup-watcher | test step completed
2026-03-14 09:56:26.803825 | controller |       0-cleanup-watcher'
2026-03-14 09:56:26.803829 | controller |     - '    logger.go:42: 09:49:19 | watcher-tls/1-deploy-with-tlse | starting test step
2026-03-14 09:56:26.803833 | controller |       1-deploy-with-tlse'
2026-03-14 09:56:26.803838 | controller |     - '    logger.go:42: 09:49:19 | watcher-tls/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-internal-svc
2026-03-14 09:56:26.803842 | controller |       created'
2026-03-14 09:56:26.803846 | controller |     - '    logger.go:42: 09:49:19 | watcher-tls/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-public-svc
2026-03-14 09:56:26.803851 | controller |       created'
2026-03-14 09:56:26.803855 | controller |     - '    logger.go:42: 09:49:19 | watcher-tls/1-deploy-with-tlse | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:56:26.803859 | controller |       created'
2026-03-14 09:56:26.803864 | controller |     - '    logger.go:42: 09:49:19 | watcher-tls/1-deploy-with-tlse | running command:
2026-03-14 09:56:26.803870 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.803874 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:56:26.803878 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.803882 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.803887 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.803891 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.803895 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.803899 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.803904 | controller |       ]'
2026-03-14 09:56:26.803908 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64
2026-03-14 09:56:26.803912 | controller |       -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:56:26.803916 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64
2026-03-14 09:56:26.803921 | controller |       -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2
2026-03-14 09:56:26.803925 | controller |       ]'
2026-03-14 09:56:26.803935 | controller |     - '        # check that both endpoints have https set'
2026-03-14 09:56:26.803940 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.803944 | controller |       infra-optim | [ $(grep -c https) == 2 ]'
2026-03-14 09:56:26.803948 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.803953 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.803957 | controller |       == "" ]; then'
2026-03-14 09:56:26.803961 | controller |     - '            exit 0'
2026-03-14 09:56:26.803965 | controller |     - '        fi'
2026-03-14 09:56:26.803970 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.803974 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.803978 | controller |     - '        counter=0'
2026-03-14 09:56:26.803982 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.803986 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.803991 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.803995 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.803999 | controller |     - '            fi'
2026-03-14 09:56:26.804003 | controller |     - '        done'
2026-03-14 09:56:26.804008 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.804012 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.804016 | controller |     - '          exit 1'
2026-03-14 09:56:26.804020 | controller |     - '        else'
2026-03-14 09:56:26.804025 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.804029 | controller |     - '        fi'
2026-03-14 09:56:26.804033 | controller |     - '        ]'
2026-03-14 09:56:26.804037 | controller |     - '    logger.go:42: 09:49:19 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.804042 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:56:26.804048 | controller |     - '    logger.go:42: 09:49:19 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.804052 | controller |     - '    logger.go:42: 09:49:21 | watcher-tls/1-deploy-with-tlse | + ''['' 0 == 1 '']'''
2026-03-14 09:56:26.804056 | controller |     - '    logger.go:42: 09:49:23 | watcher-tls/1-deploy-with-tlse | running command:
2026-03-14 09:56:26.804060 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.804065 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:56:26.804069 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.804075 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.804080 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.804084 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.804091 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.804095 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.804100 | controller |       ]'
2026-03-14 09:56:26.804104 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64
2026-03-14 09:56:26.804108 | controller |       -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:56:26.804113 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64
2026-03-14 09:56:26.804117 | controller |       -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2
2026-03-14 09:56:26.804121 | controller |       ]'
2026-03-14 09:56:26.804126 | controller |     - '        # check that both endpoints have https set'
2026-03-14 09:56:26.804130 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.804134 | controller |       infra-optim | [ $(grep -c https) == 2 ]'
2026-03-14 09:56:26.804138 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.804143 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.804147 | controller |       == "" ]; then'
2026-03-14 09:56:26.804151 | controller |     - '            exit 0'
2026-03-14 09:56:26.804155 | controller |     - '        fi'
2026-03-14 09:56:26.804160 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.804164 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.804168 | controller |     - '        counter=0'
2026-03-14 09:56:26.804172 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.804177 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.804181 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.804185 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.804189 | controller |     - '            fi'
2026-03-14 09:56:26.804193 | controller |     - '        done'
2026-03-14 09:56:26.804198 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.804202 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.804206 | controller |     - '          exit 1'
2026-03-14 09:56:26.804210 | controller |     - '        else'
2026-03-14 09:56:26.804215 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.804219 | controller |     - '        fi'
2026-03-14 09:56:26.804223 | controller |     - '        ]'
2026-03-14 09:56:26.804227 | controller |     - '    logger.go:42: 09:49:23 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.804232 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:56:26.804236 | controller |     - '    logger.go:42: 09:49:23 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.804240 | controller |     - '    logger.go:42: 09:49:25 | watcher-tls/1-deploy-with-tlse | + ''['' 0 == 1 '']'''
2026-03-14 09:56:26.804250 | controller |     - '    logger.go:42: 09:49:26 | watcher-tls/1-deploy-with-tlse | running command:
2026-03-14 09:56:26.804255 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.804259 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:56:26.804263 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.804268 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.804272 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.804276 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.804283 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.804287 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.804291 | controller |       ]'
2026-03-14 09:56:26.804295 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64
2026-03-14 09:56:26.804300 | controller |       -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:56:26.804304 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64
2026-03-14 09:56:26.804308 | controller |       -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2
2026-03-14 09:56:26.804312 | controller |       ]'
2026-03-14 09:56:26.804317 | controller |     - '        # check that both endpoints have https set'
2026-03-14 09:56:26.804323 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.804327 | controller |       infra-optim | [ $(grep -c https) == 2 ]'
2026-03-14 09:56:26.804332 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.804336 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.804341 | controller |       == "" ]; then'
2026-03-14 09:56:26.804346 | controller |     - '            exit 0'
2026-03-14 09:56:26.804351 | controller |     - '        fi'
2026-03-14 09:56:26.804355 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.804361 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.804366 | controller |     - '        counter=0'
2026-03-14 09:56:26.804370 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.804375 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.804379 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.804383 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.804388 | controller |     - '            fi'
2026-03-14 09:56:26.804392 | controller |     - '        done'
2026-03-14 09:56:26.804396 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.804400 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.804405 | controller |     - '          exit 1'
2026-03-14 09:56:26.804409 | controller |     - '        else'
2026-03-14 09:56:26.804413 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.804417 | controller |     - '        fi'
2026-03-14 09:56:26.804422 | controller |     - '        ]'
2026-03-14 09:56:26.804426 | controller |     - '    logger.go:42: 09:49:26 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.804430 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:56:26.804434 | controller |     - '    logger.go:42: 09:49:26 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.804439 | controller |     - '    logger.go:42: 09:49:28 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.804443 | controller |     - '    logger.go:42: 09:49:28 | watcher-tls/1-deploy-with-tlse | ++ grep watcher'
2026-03-14 09:56:26.804447 | controller |     - '    logger.go:42: 09:49:28 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.804451 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:56:26.804455 | controller |     - '    logger.go:42: 09:49:28 | watcher-tls/1-deploy-with-tlse | ++ awk ''{print $1}'''
2026-03-14 09:56:26.804460 | controller |     - '    logger.go:42: 09:49:30 | watcher-tls/1-deploy-with-tlse | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa'
2026-03-14 09:56:26.804464 | controller |     - '    logger.go:42: 09:49:30 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.804468 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.804472 | controller |     - '    logger.go:42: 09:49:30 | watcher-tls/1-deploy-with-tlse | + ''['' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.804478 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa '']'''
2026-03-14 09:56:26.804482 | controller |     - '    logger.go:42: 09:49:30 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.804487 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.804491 | controller |     - '    logger.go:42: 09:49:30 | watcher-tls/1-deploy-with-tlse | + ''['' -n '''' '']'''
2026-03-14 09:56:26.804495 | controller |     - '    logger.go:42: 09:49:31 | watcher-tls/1-deploy-with-tlse | running command:
2026-03-14 09:56:26.804499 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.804504 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:56:26.804508 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.804512 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.804516 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.804526 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.804531 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.804535 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.804539 | controller |       ]'
2026-03-14 09:56:26.804544 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64
2026-03-14 09:56:26.804548 | controller |       -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:56:26.804552 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64
2026-03-14 09:56:26.804556 | controller |       -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2
2026-03-14 09:56:26.804561 | controller |       ]'
2026-03-14 09:56:26.804565 | controller |     - '        # check that both endpoints have https set'
2026-03-14 09:56:26.804569 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.804573 | controller |       infra-optim | [ $(grep -c https) == 2 ]'
2026-03-14 09:56:26.804577 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.804582 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.804586 | controller |       == "" ]; then'
2026-03-14 09:56:26.804590 | controller |     - '            exit 0'
2026-03-14 09:56:26.804605 | controller |     - '        fi'
2026-03-14 09:56:26.804611 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.804615 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.804619 | controller |     - '        counter=0'
2026-03-14 09:56:26.804624 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.804628 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.804632 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.804636 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.804641 | controller |     - '            fi'
2026-03-14 09:56:26.804645 | controller |     - '        done'
2026-03-14 09:56:26.804649 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.804653 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.804657 | controller |     - '          exit 1'
2026-03-14 09:56:26.804662 | controller |     - '        else'
2026-03-14 09:56:26.804666 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.804670 | controller |     - '        fi'
2026-03-14 09:56:26.804674 | controller |     - '        ]'
2026-03-14 09:56:26.804679 | controller |     - '    logger.go:42: 09:49:31 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.804683 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:56:26.804689 | controller |     - '    logger.go:42: 09:49:31 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.804693 | controller |     - '    logger.go:42: 09:49:33 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.804698 | controller |     - '    logger.go:42: 09:49:33 | watcher-tls/1-deploy-with-tlse | ++ grep watcher'
2026-03-14 09:56:26.804702 | controller |     - '    logger.go:42: 09:49:33 | watcher-tls/1-deploy-with-tlse | ++ awk ''{print $1}'''
2026-03-14 09:56:26.804706 | controller |     - '    logger.go:42: 09:49:33 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.804710 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:56:26.804715 | controller |     - '    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa'
2026-03-14 09:56:26.804719 | controller |     - '    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.804723 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.804727 | controller |     - '    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | + ''['' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.804732 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa '']'''
2026-03-14 09:56:26.804736 | controller |     - '    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.804740 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.804744 | controller |     - '    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | + ''['' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.804759 | controller |       '']'''
2026-03-14 09:56:26.804763 | controller |     - '    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | ++ base64 -d'
2026-03-14 09:56:26.804768 | controller |     - '    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1'
2026-03-14 09:56:26.804772 | controller |     - '    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.804776 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}'''
2026-03-14 09:56:26.804780 | controller |     - '    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.804785 | controller |     - '    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.804789 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:56:26.804793 | controller |     - '    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''cafile
2026-03-14 09:56:26.804797 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'''
2026-03-14 09:56:26.804808 | controller |     - '    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | ++ base64 -d'
2026-03-14 09:56:26.804812 | controller |     - '    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']'''
2026-03-14 09:56:26.804817 | controller |     - '    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.804821 | controller |       openstackclient -- openstack endpoint list'
2026-03-14 09:56:26.804825 | controller |     - '    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | + grep infra-optim'
2026-03-14 09:56:26.804829 | controller |     - '    logger.go:42: 09:49:36 | watcher-tls/1-deploy-with-tlse | ++ grep -c https'
2026-03-14 09:56:26.804834 | controller |     - '    logger.go:42: 09:49:38 | watcher-tls/1-deploy-with-tlse | + ''['' 0 == 2 '']'''
2026-03-14 09:56:26.804838 | controller |     - '    logger.go:42: 09:49:39 | watcher-tls/1-deploy-with-tlse | running command:
2026-03-14 09:56:26.804842 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.804846 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:56:26.804850 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.804855 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.804862 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.804867 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.804871 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.804875 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.804880 | controller |       ]'
2026-03-14 09:56:26.804884 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64
2026-03-14 09:56:26.804888 | controller |       -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:56:26.804892 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64
2026-03-14 09:56:26.804896 | controller |       -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2
2026-03-14 09:56:26.804901 | controller |       ]'
2026-03-14 09:56:26.804905 | controller |     - '        # check that both endpoints have https set'
2026-03-14 09:56:26.804909 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.804914 | controller |       infra-optim | [ $(grep -c https) == 2 ]'
2026-03-14 09:56:26.804918 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.804922 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.804926 | controller |       == "" ]; then'
2026-03-14 09:56:26.804931 | controller |     - '            exit 0'
2026-03-14 09:56:26.804935 | controller |     - '        fi'
2026-03-14 09:56:26.804939 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.804943 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.804948 | controller |     - '        counter=0'
2026-03-14 09:56:26.804952 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.804956 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.804960 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.804965 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.804969 | controller |     - '            fi'
2026-03-14 09:56:26.804973 | controller |     - '        done'
2026-03-14 09:56:26.804977 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.804982 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.804986 | controller |     - '          exit 1'
2026-03-14 09:56:26.804990 | controller |     - '        else'
2026-03-14 09:56:26.804994 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.804999 | controller |     - '        fi'
2026-03-14 09:56:26.805003 | controller |     - '        ]'
2026-03-14 09:56:26.805007 | controller |     - '    logger.go:42: 09:49:39 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.805011 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:56:26.805016 | controller |     - '    logger.go:42: 09:49:39 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.805020 | controller |     - '    logger.go:42: 09:49:42 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.805024 | controller |     - '    logger.go:42: 09:49:42 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.805028 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:56:26.805034 | controller |     - '    logger.go:42: 09:49:42 | watcher-tls/1-deploy-with-tlse | ++ grep watcher'
2026-03-14 09:56:26.805039 | controller |     - '    logger.go:42: 09:49:42 | watcher-tls/1-deploy-with-tlse | ++ awk ''{print $1}'''
2026-03-14 09:56:26.805043 | controller |     - '    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa'
2026-03-14 09:56:26.805047 | controller |     - '    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.805053 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.805057 | controller |     - '    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | + ''['' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.805061 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa '']'''
2026-03-14 09:56:26.805066 | controller |     - '    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.805076 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.805081 | controller |     - '    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | + ''['' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.805086 | controller |       '']'''
2026-03-14 09:56:26.805091 | controller |     - '    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | ++ base64 -d'
2026-03-14 09:56:26.805095 | controller |     - '    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.805099 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}'''
2026-03-14 09:56:26.805104 | controller |     - '    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1'
2026-03-14 09:56:26.805108 | controller |     - '    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.805112 | controller |     - '    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''cafile
2026-03-14 09:56:26.805116 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'''
2026-03-14 09:56:26.805120 | controller |     - '    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | ++ base64 -d'
2026-03-14 09:56:26.805125 | controller |     - '    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.805129 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:56:26.805133 | controller |     - '    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']'''
2026-03-14 09:56:26.805137 | controller |     - '    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | + grep infra-optim'
2026-03-14 09:56:26.805142 | controller |     - '    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.805146 | controller |       openstackclient -- openstack endpoint list'
2026-03-14 09:56:26.805150 | controller |     - '    logger.go:42: 09:49:44 | watcher-tls/1-deploy-with-tlse | ++ grep -c https'
2026-03-14 09:56:26.805154 | controller |     - '    logger.go:42: 09:49:47 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']'''
2026-03-14 09:56:26.805158 | controller |     - '    logger.go:42: 09:49:47 | watcher-tls/1-deploy-with-tlse | ++ oc get pods -n
2026-03-14 09:56:26.805163 | controller |       openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:56:26.805167 | controller |     - '    logger.go:42: 09:49:47 | watcher-tls/1-deploy-with-tlse | + ''['' '''' == ''''
2026-03-14 09:56:26.805171 | controller |       '']'''
2026-03-14 09:56:26.805176 | controller |     - '    logger.go:42: 09:49:47 | watcher-tls/1-deploy-with-tlse | + exit 0'
2026-03-14 09:56:26.805180 | controller |     - '    logger.go:42: 09:49:48 | watcher-tls/1-deploy-with-tlse | running command:
2026-03-14 09:56:26.805184 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.805188 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:56:26.805192 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.805197 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.805201 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.805205 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.805209 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.805213 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.805219 | controller |       ]'
2026-03-14 09:56:26.805224 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64
2026-03-14 09:56:26.805228 | controller |       -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:56:26.805232 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64
2026-03-14 09:56:26.805236 | controller |       -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2
2026-03-14 09:56:26.805241 | controller |       ]'
2026-03-14 09:56:26.805245 | controller |     - '        # check that both endpoints have https set'
2026-03-14 09:56:26.805249 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.805253 | controller |       infra-optim | [ $(grep -c https) == 2 ]'
2026-03-14 09:56:26.805258 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.805262 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.805266 | controller |       == "" ]; then'
2026-03-14 09:56:26.805270 | controller |     - '            exit 0'
2026-03-14 09:56:26.805275 | controller |     - '        fi'
2026-03-14 09:56:26.805279 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.805283 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.805287 | controller |     - '        counter=0'
2026-03-14 09:56:26.805292 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.805296 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.805300 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.805304 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.805309 | controller |     - '            fi'
2026-03-14 09:56:26.805313 | controller |     - '        done'
2026-03-14 09:56:26.805317 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.805321 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.805325 | controller |     - '          exit 1'
2026-03-14 09:56:26.805330 | controller |     - '        else'
2026-03-14 09:56:26.805339 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.805344 | controller |     - '        fi'
2026-03-14 09:56:26.805348 | controller |     - '        ]'
2026-03-14 09:56:26.805352 | controller |     - '    logger.go:42: 09:49:48 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.805357 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:56:26.805361 | controller |     - '    logger.go:42: 09:49:48 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.805365 | controller |     - '    logger.go:42: 09:49:50 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.805369 | controller |     - '    logger.go:42: 09:49:50 | watcher-tls/1-deploy-with-tlse | ++ grep watcher'
2026-03-14 09:56:26.805373 | controller |     - '    logger.go:42: 09:49:50 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.805378 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:56:26.805384 | controller |     - '    logger.go:42: 09:49:50 | watcher-tls/1-deploy-with-tlse | ++ awk ''{print $1}'''
2026-03-14 09:56:26.805388 | controller |     - '    logger.go:42: 09:49:52 | watcher-tls/1-deploy-with-tlse | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa'
2026-03-14 09:56:26.805392 | controller |     - '    logger.go:42: 09:49:52 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.805396 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.805401 | controller |     - '    logger.go:42: 09:49:52 | watcher-tls/1-deploy-with-tlse | + ''['' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.805405 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa '']'''
2026-03-14 09:56:26.805409 | controller |     - '    logger.go:42: 09:49:52 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.805415 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.805419 | controller |     - '    logger.go:42: 09:49:52 | watcher-tls/1-deploy-with-tlse | + ''['' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.805425 | controller |       '']'''
2026-03-14 09:56:26.805429 | controller |     - '    logger.go:42: 09:49:52 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1'
2026-03-14 09:56:26.805434 | controller |     - '    logger.go:42: 09:49:52 | watcher-tls/1-deploy-with-tlse | ++ base64 -d'
2026-03-14 09:56:26.805438 | controller |     - '    logger.go:42: 09:49:52 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.805442 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}'''
2026-03-14 09:56:26.805446 | controller |     - '    logger.go:42: 09:49:53 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.805450 | controller |     - '    logger.go:42: 09:49:53 | watcher-tls/1-deploy-with-tlse | ++ base64 -d'
2026-03-14 09:56:26.805455 | controller |     - '    logger.go:42: 09:49:53 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.805459 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:56:26.805463 | controller |     - '    logger.go:42: 09:49:53 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''cafile
2026-03-14 09:56:26.805467 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'''
2026-03-14 09:56:26.805471 | controller |     - '    logger.go:42: 09:49:53 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']'''
2026-03-14 09:56:26.805475 | controller |     - '    logger.go:42: 09:49:53 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.805480 | controller |       openstackclient -- openstack endpoint list'
2026-03-14 09:56:26.805484 | controller |     - '    logger.go:42: 09:49:53 | watcher-tls/1-deploy-with-tlse | + grep infra-optim'
2026-03-14 09:56:26.805488 | controller |     - '    logger.go:42: 09:49:53 | watcher-tls/1-deploy-with-tlse | ++ grep -c https'
2026-03-14 09:56:26.805492 | controller |     - '    logger.go:42: 09:49:55 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']'''
2026-03-14 09:56:26.805496 | controller |     - '    logger.go:42: 09:49:55 | watcher-tls/1-deploy-with-tlse | ++ oc get pods -n
2026-03-14 09:56:26.805501 | controller |       openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:56:26.805505 | controller |     - '    logger.go:42: 09:49:55 | watcher-tls/1-deploy-with-tlse | + ''['' '''' == ''''
2026-03-14 09:56:26.805509 | controller |       '']'''
2026-03-14 09:56:26.805513 | controller |     - '    logger.go:42: 09:49:55 | watcher-tls/1-deploy-with-tlse | + exit 0'
2026-03-14 09:56:26.805518 | controller |     - '    logger.go:42: 09:49:55 | watcher-tls/1-deploy-with-tlse | test step completed
2026-03-14 09:56:26.805522 | controller |       1-deploy-with-tlse'
2026-03-14 09:56:26.805526 | controller |     - '    logger.go:42: 09:49:55 | watcher-tls/2-patch-mtls | starting test step 2-patch-mtls'
2026-03-14 09:56:26.805530 | controller |     - '    logger.go:42: 09:49:55 | watcher-tls/2-patch-mtls | running command: [sh -c
2026-03-14 09:56:26.805534 | controller |       set -euxo pipefail'
2026-03-14 09:56:26.805539 | controller |     - '        oc patch oscp -n $NAMESPACE openstack --type=''json'' -p=''[{"op": "replace",
2026-03-14 09:56:26.805543 | controller |       "path": "/spec/memcached/templates/memcached/tls/mtls/sslVerifyMode", "value": "Request"}]'''
2026-03-14 09:56:26.805547 | controller |     - '        ]'
2026-03-14 09:56:26.805551 | controller |     - '    logger.go:42: 09:49:55 | watcher-tls/2-patch-mtls | + oc patch oscp -n watcher-kuttl-default
2026-03-14 09:56:26.805561 | controller |       openstack --type=json ''-p=[{"op": "replace", "path": "/spec/memcached/templates/memcached/tls/mtls/sslVerifyMode",
2026-03-14 09:56:26.805566 | controller |       "value": "Request"}]'''
2026-03-14 09:56:26.805570 | controller |     - '    logger.go:42: 09:49:55 | watcher-tls/2-patch-mtls | openstackcontrolplane.core.openstack.org/openstack
2026-03-14 09:56:26.805574 | controller |       patched'
2026-03-14 09:56:26.805579 | controller |     - '    logger.go:42: 09:49:55 | watcher-tls/2-patch-mtls | running command: [sh -c
2026-03-14 09:56:26.805586 | controller |       set -euxo pipefail'
2026-03-14 09:56:26.805590 | controller |     - '        '
2026-03-14 09:56:26.805604 | controller |     - '        oc project ${NAMESPACE}'
2026-03-14 09:56:26.805610 | controller |     - '        # Get pod names for each watcher service'
2026-03-14 09:56:26.805614 | controller |     - '        APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:56:26.805619 | controller |     - '        APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:56:26.805623 | controller |     - '        DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:56:26.805627 | controller |     - '        '
2026-03-14 09:56:26.805631 | controller |     - '        # Verify memcached mTLS config parameters in watcher-api config'
2026-03-14 09:56:26.805635 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.805640 | controller |     - '          echo "Checking watcher-api config..."'
2026-03-14 09:56:26.805644 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805648 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:56:26.805652 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805657 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.805661 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805665 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.805669 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805673 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:56:26.805678 | controller |     - '        '
2026-03-14 09:56:26.805682 | controller |     - '          # Verify mTLS config parameters in memcached backend config'
2026-03-14 09:56:26.805686 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805690 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:56:26.805695 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805699 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.805703 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805707 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.805712 | controller |     - '        else'
2026-03-14 09:56:26.805716 | controller |     - '          exit 1'
2026-03-14 09:56:26.805720 | controller |     - '        fi'
2026-03-14 09:56:26.805724 | controller |     - '        '
2026-03-14 09:56:26.805729 | controller |     - '        # Verify memcached mTLS config parameters in watcher-applier config'
2026-03-14 09:56:26.805733 | controller |     - '        if [ -n "${APPLIERPOD}" ]; then'
2026-03-14 09:56:26.805737 | controller |     - '          echo "Checking watcher-applier config..."'
2026-03-14 09:56:26.805741 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805746 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:56:26.805750 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805754 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.805760 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805765 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.805769 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805776 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:56:26.805780 | controller |     - '        '
2026-03-14 09:56:26.805784 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805789 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:56:26.805793 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805797 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.805801 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805806 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.805810 | controller |     - '        else'
2026-03-14 09:56:26.805814 | controller |     - '          exit 1'
2026-03-14 09:56:26.805818 | controller |     - '        fi'
2026-03-14 09:56:26.805823 | controller |     - '        '
2026-03-14 09:56:26.805827 | controller |     - '        # Verify memcached mTLS config parameters in watcher-decision-engine config'
2026-03-14 09:56:26.805838 | controller |     - '        if [ -n "${DECISIONENGINEPOD}" ]; then'
2026-03-14 09:56:26.805842 | controller |     - '          echo "Checking watcher-decision-engine config..."'
2026-03-14 09:56:26.805847 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805851 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:56:26.805855 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805860 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.805864 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805868 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.805872 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805877 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:56:26.805881 | controller |     - '        '
2026-03-14 09:56:26.805887 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805891 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:56:26.805895 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805899 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.805904 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.805908 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.805912 | controller |     - '        else'
2026-03-14 09:56:26.805916 | controller |     - '          exit 1'
2026-03-14 09:56:26.805921 | controller |     - '        fi'
2026-03-14 09:56:26.805925 | controller |     - '        ]'
2026-03-14 09:56:26.805929 | controller |     - '    logger.go:42: 09:49:55 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default'
2026-03-14 09:56:26.805933 | controller |     - '    logger.go:42: 09:49:55 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default"
2026-03-14 09:56:26.805938 | controller |       on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.805942 | controller |     - '    logger.go:42: 09:49:55 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api
2026-03-14 09:56:26.805946 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:56:26.805950 | controller |     - '    logger.go:42: 09:49:55 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.805956 | controller |     - '    logger.go:42: 09:49:55 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier
2026-03-14 09:56:26.805960 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:56:26.805965 | controller |     - '    logger.go:42: 09:49:56 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0'
2026-03-14 09:56:26.805969 | controller |     - '    logger.go:42: 09:49:56 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine
2026-03-14 09:56:26.805973 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:56:26.805977 | controller |     - '    logger.go:42: 09:49:56 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.805982 | controller |     - '    logger.go:42: 09:49:56 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0
2026-03-14 09:56:26.805987 | controller |       '']'''
2026-03-14 09:56:26.805993 | controller |     - '    logger.go:42: 09:49:56 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api
2026-03-14 09:56:26.805997 | controller |       config...'''
2026-03-14 09:56:26.806002 | controller |     - '    logger.go:42: 09:49:56 | watcher-tls/2-patch-mtls | Checking watcher-api config...'
2026-03-14 09:56:26.806007 | controller |     - '    logger.go:42: 09:49:56 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile
2026-03-14 09:56:26.806011 | controller |       = /etc/pki/tls/certs/mtls.crt '''
2026-03-14 09:56:26.806015 | controller |     - '    logger.go:42: 09:49:56 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.806019 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.806024 | controller |     - '    logger.go:42: 09:49:58 | watcher-tls/2-patch-mtls | error: Internal error occurred:
2026-03-14 09:56:26.806028 | controller |       error executing command in container: container is not created or running'
2026-03-14 09:56:26.806032 | controller |     - '    logger.go:42: 09:49:58 | watcher-tls/2-patch-mtls | ++ echo'
2026-03-14 09:56:26.806036 | controller |     - '    logger.go:42: 09:49:58 | watcher-tls/2-patch-mtls | + ''['' 0 == 1 '']'''
2026-03-14 09:56:26.806040 | controller |     - '    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | running command: [sh -c
2026-03-14 09:56:26.806045 | controller |       set -euxo pipefail'
2026-03-14 09:56:26.806051 | controller |     - '        '
2026-03-14 09:56:26.806055 | controller |     - '        oc project ${NAMESPACE}'
2026-03-14 09:56:26.806059 | controller |     - '        # Get pod names for each watcher service'
2026-03-14 09:56:26.806063 | controller |     - '        APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:56:26.806067 | controller |     - '        APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:56:26.806077 | controller |     - '        DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:56:26.806082 | controller |     - '        '
2026-03-14 09:56:26.806086 | controller |     - '        # Verify memcached mTLS config parameters in watcher-api config'
2026-03-14 09:56:26.806091 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.806095 | controller |     - '          echo "Checking watcher-api config..."'
2026-03-14 09:56:26.806099 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806103 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:56:26.806107 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806112 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.806116 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806120 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.806124 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806130 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:56:26.806135 | controller |     - '        '
2026-03-14 09:56:26.806139 | controller |     - '          # Verify mTLS config parameters in memcached backend config'
2026-03-14 09:56:26.806143 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806147 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:56:26.806152 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806156 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.806160 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806165 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.806169 | controller |     - '        else'
2026-03-14 09:56:26.806173 | controller |     - '          exit 1'
2026-03-14 09:56:26.806177 | controller |     - '        fi'
2026-03-14 09:56:26.806182 | controller |     - '        '
2026-03-14 09:56:26.806186 | controller |     - '        # Verify memcached mTLS config parameters in watcher-applier config'
2026-03-14 09:56:26.806190 | controller |     - '        if [ -n "${APPLIERPOD}" ]; then'
2026-03-14 09:56:26.806195 | controller |     - '          echo "Checking watcher-applier config..."'
2026-03-14 09:56:26.806199 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806203 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:56:26.806208 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806212 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.806216 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806221 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.806225 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806229 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:56:26.806233 | controller |     - '        '
2026-03-14 09:56:26.806238 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806242 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:56:26.806246 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806251 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.806255 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806259 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.806263 | controller |     - '        else'
2026-03-14 09:56:26.806268 | controller |     - '          exit 1'
2026-03-14 09:56:26.806272 | controller |     - '        fi'
2026-03-14 09:56:26.806276 | controller |     - '        '
2026-03-14 09:56:26.806281 | controller |     - '        # Verify memcached mTLS config parameters in watcher-decision-engine config'
2026-03-14 09:56:26.806285 | controller |     - '        if [ -n "${DECISIONENGINEPOD}" ]; then'
2026-03-14 09:56:26.806289 | controller |     - '          echo "Checking watcher-decision-engine config..."'
2026-03-14 09:56:26.806294 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806298 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:56:26.806302 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806308 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.806313 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806323 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.806327 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806332 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:56:26.806336 | controller |     - '        '
2026-03-14 09:56:26.806340 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806345 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:56:26.806349 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806353 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.806357 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806362 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.806366 | controller |     - '        else'
2026-03-14 09:56:26.806370 | controller |     - '          exit 1'
2026-03-14 09:56:26.806375 | controller |     - '        fi'
2026-03-14 09:56:26.806379 | controller |     - '        ]'
2026-03-14 09:56:26.806383 | controller |     - '    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default'
2026-03-14 09:56:26.806388 | controller |     - '    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default"
2026-03-14 09:56:26.806392 | controller |       on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.806396 | controller |     - '    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api
2026-03-14 09:56:26.806401 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:56:26.806405 | controller |     - '    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.806409 | controller |     - '    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier
2026-03-14 09:56:26.806414 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:56:26.806418 | controller |     - '    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0'
2026-03-14 09:56:26.806422 | controller |     - '    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine
2026-03-14 09:56:26.806426 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:56:26.806431 | controller |     - '    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.806435 | controller |     - '    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0
2026-03-14 09:56:26.806439 | controller |       '']'''
2026-03-14 09:56:26.806444 | controller |     - '    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api
2026-03-14 09:56:26.806448 | controller |       config...'''
2026-03-14 09:56:26.806452 | controller |     - '    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | Checking watcher-api config...'
2026-03-14 09:56:26.806456 | controller |     - '    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile
2026-03-14 09:56:26.806461 | controller |       = /etc/pki/tls/certs/mtls.crt '''
2026-03-14 09:56:26.806465 | controller |     - '    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.806469 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.806474 | controller |     - '    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | error: unable to upgrade
2026-03-14 09:56:26.806478 | controller |       connection: container not found ("watcher-api")'
2026-03-14 09:56:26.806484 | controller |     - '    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | ++ echo'
2026-03-14 09:56:26.806488 | controller |     - '    logger.go:42: 09:49:59 | watcher-tls/2-patch-mtls | + ''['' 0 == 1 '']'''
2026-03-14 09:56:26.806492 | controller |     - '    logger.go:42: 09:50:00 | watcher-tls/2-patch-mtls | running command: [sh -c
2026-03-14 09:56:26.806497 | controller |       set -euxo pipefail'
2026-03-14 09:56:26.806501 | controller |     - '        '
2026-03-14 09:56:26.806505 | controller |     - '        oc project ${NAMESPACE}'
2026-03-14 09:56:26.806510 | controller |     - '        # Get pod names for each watcher service'
2026-03-14 09:56:26.806514 | controller |     - '        APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:56:26.806518 | controller |     - '        APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:56:26.806522 | controller |     - '        DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:56:26.806527 | controller |     - '        '
2026-03-14 09:56:26.806531 | controller |     - '        # Verify memcached mTLS config parameters in watcher-api config'
2026-03-14 09:56:26.806535 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.806540 | controller |     - '          echo "Checking watcher-api config..."'
2026-03-14 09:56:26.806544 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806548 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:56:26.806552 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806557 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.806561 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806571 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.806576 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806580 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:56:26.806586 | controller |     - '        '
2026-03-14 09:56:26.806591 | controller |     - '          # Verify mTLS config parameters in memcached backend config'
2026-03-14 09:56:26.806605 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806611 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:56:26.806615 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806620 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.806624 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806628 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.806633 | controller |     - '        else'
2026-03-14 09:56:26.806637 | controller |     - '          exit 1'
2026-03-14 09:56:26.806641 | controller |     - '        fi'
2026-03-14 09:56:26.806646 | controller |     - '        '
2026-03-14 09:56:26.806652 | controller |     - '        # Verify memcached mTLS config parameters in watcher-applier config'
2026-03-14 09:56:26.806656 | controller |     - '        if [ -n "${APPLIERPOD}" ]; then'
2026-03-14 09:56:26.806661 | controller |     - '          echo "Checking watcher-applier config..."'
2026-03-14 09:56:26.806665 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806669 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:56:26.806674 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806678 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.806684 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806688 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.806693 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806697 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:56:26.806701 | controller |     - '        '
2026-03-14 09:56:26.806705 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806710 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:56:26.806714 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806718 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.806723 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806727 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.806731 | controller |     - '        else'
2026-03-14 09:56:26.806736 | controller |     - '          exit 1'
2026-03-14 09:56:26.806740 | controller |     - '        fi'
2026-03-14 09:56:26.806747 | controller |     - '        '
2026-03-14 09:56:26.806752 | controller |     - '        # Verify memcached mTLS config parameters in watcher-decision-engine config'
2026-03-14 09:56:26.806757 | controller |     - '        if [ -n "${DECISIONENGINEPOD}" ]; then'
2026-03-14 09:56:26.806762 | controller |     - '          echo "Checking watcher-decision-engine config..."'
2026-03-14 09:56:26.806766 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806770 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:56:26.806775 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806779 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.806783 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806788 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.806792 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806796 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:56:26.806801 | controller |     - '        '
2026-03-14 09:56:26.806805 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806809 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:56:26.806813 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806818 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.806828 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.806833 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.806838 | controller |     - '        else'
2026-03-14 09:56:26.806842 | controller |     - '          exit 1'
2026-03-14 09:56:26.806846 | controller |     - '        fi'
2026-03-14 09:56:26.806851 | controller |     - '        ]'
2026-03-14 09:56:26.806855 | controller |     - '    logger.go:42: 09:50:00 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default'
2026-03-14 09:56:26.806859 | controller |     - '    logger.go:42: 09:50:00 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default"
2026-03-14 09:56:26.806865 | controller |       on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.806870 | controller |     - '    logger.go:42: 09:50:00 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api
2026-03-14 09:56:26.806874 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:56:26.806878 | controller |     - '    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.806883 | controller |     - '    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier
2026-03-14 09:56:26.806887 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:56:26.806891 | controller |     - '    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0'
2026-03-14 09:56:26.806896 | controller |     - '    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine
2026-03-14 09:56:26.806900 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:56:26.806904 | controller |     - '    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.806909 | controller |     - '    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0
2026-03-14 09:56:26.806913 | controller |       '']'''
2026-03-14 09:56:26.806917 | controller |     - '    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api
2026-03-14 09:56:26.806921 | controller |       config...'''
2026-03-14 09:56:26.806926 | controller |     - '    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | Checking watcher-api config...'
2026-03-14 09:56:26.806930 | controller |     - '    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile
2026-03-14 09:56:26.806934 | controller |       = /etc/pki/tls/certs/mtls.crt '''
2026-03-14 09:56:26.806939 | controller |     - '    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.806943 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.806947 | controller |     - '    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.806951 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.806956 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.806960 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.806965 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.806969 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.806973 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.806977 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.806982 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.806986 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.806990 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.806995 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.806999 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.807003 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.807008 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.807012 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.807018 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.807022 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.807026 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.807031 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.807035 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.807062 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.807067 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.807071 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.807076 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.807080 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.807084 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.807089 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.807093 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.807097 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.807101 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.807106 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.807110 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.807114 | controller |     - '    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.807119 | controller |     - '    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile
2026-03-14 09:56:26.807123 | controller |       = /etc/pki/tls/private/mtls.key'''
2026-03-14 09:56:26.807130 | controller |     - '    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.807134 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.807138 | controller |     - '    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.807143 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.807147 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.807151 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.807156 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.807160 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.807164 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.807168 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.807173 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.807177 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.807183 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.807187 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.807192 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.807196 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.807200 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.807205 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.807209 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.807215 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.807219 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.807223 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.807228 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.807232 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.807236 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.807240 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.807245 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.807249 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.807259 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.807263 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.807268 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.807272 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.807276 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.807281 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.807285 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.807289 | controller |     - '    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.807293 | controller |     - '    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile
2026-03-14 09:56:26.807298 | controller |       = /etc/pki/tls/certs/mtls-ca.crt'''
2026-03-14 09:56:26.807302 | controller |     - '    logger.go:42: 09:50:01 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.807306 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.807310 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.807315 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.807319 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.807323 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.807327 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.807333 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.807338 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.807342 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.807346 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.807350 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.807354 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.807359 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.807363 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.807367 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.807371 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.807375 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.807380 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.807384 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.807388 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.807392 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.807397 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.807401 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.807405 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.807409 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.807413 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.807419 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.807424 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.807428 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.807432 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.807436 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.807446 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.807451 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.807455 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.807459 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.807464 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled
2026-03-14 09:56:26.807468 | controller |       = true'''
2026-03-14 09:56:26.807472 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.807476 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.807482 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.807486 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.807490 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.807495 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.807499 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.807503 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.807509 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.807513 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.807518 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.807522 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.807526 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.807530 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.807534 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.807539 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.807543 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.807547 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.807551 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.807556 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.807560 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.807564 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.807568 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.807572 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.807577 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.807581 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.807585 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.807589 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.807593 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.807609 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.807614 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.807618 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.807622 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.807626 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.807633 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.807637 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.807648 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt'
2026-03-14 09:56:26.807652 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.807656 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.807663 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.807667 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.807671 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.807675 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.807680 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.807684 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.807688 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.807692 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.807697 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.807701 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.807705 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.807709 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.807713 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.807717 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.807723 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.807728 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.807732 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.807736 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.807740 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.807744 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.807749 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.807753 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.807757 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.807761 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.807765 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.807770 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.807775 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.807780 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.807784 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.807788 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.807792 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.807796 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.807801 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.807805 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.807809 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key'
2026-03-14 09:56:26.807813 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.807818 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.807822 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.807832 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.807836 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.807841 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.807848 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.807852 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.807857 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.807861 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.807865 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.807869 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.807873 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.807878 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.807882 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.807886 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.807890 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.807894 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.807899 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.807903 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.807907 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.807911 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.807915 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.807921 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.807925 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.807930 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.807934 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.807938 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.807942 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.807947 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.807951 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.807955 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.807959 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.807963 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.807969 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.807974 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.807978 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt'
2026-03-14 09:56:26.807982 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.807986 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.807991 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.807995 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.807999 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.808003 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.808014 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.808018 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.808022 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.808027 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.808031 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.808035 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.808039 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.808044 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.808048 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.808054 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.808058 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.808064 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.808068 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.808072 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.808077 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.808081 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.808085 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.808089 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.808094 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.808098 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.808102 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.808106 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.808111 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.808115 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.808119 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.808124 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.808128 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.808132 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.808136 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.808141 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.808145 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-applier-0
2026-03-14 09:56:26.808149 | controller |       '']'''
2026-03-14 09:56:26.808154 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-applier
2026-03-14 09:56:26.808158 | controller |       config...'''
2026-03-14 09:56:26.808162 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | Checking watcher-applier
2026-03-14 09:56:26.808166 | controller |       config...'
2026-03-14 09:56:26.808171 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile
2026-03-14 09:56:26.808175 | controller |       = /etc/pki/tls/certs/mtls.crt '''
2026-03-14 09:56:26.808179 | controller |     - '    logger.go:42: 09:50:02 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.808183 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.808188 | controller |     - '    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.808192 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.808196 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.808201 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.808212 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.808218 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.808222 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.808227 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.808231 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.808235 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.808240 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.808244 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.808248 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.808253 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.808257 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.808261 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.808266 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.808271 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.808276 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.808280 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.808284 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.808288 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:56:26.808293 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.808297 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.808301 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.808306 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.808310 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.808314 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.808318 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.808323 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.808327 | controller |     - '    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.808331 | controller |     - '    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile
2026-03-14 09:56:26.808336 | controller |       = /etc/pki/tls/private/mtls.key'''
2026-03-14 09:56:26.808371 | controller |     - '    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.808376 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.808381 | controller |     - '    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.808385 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.808389 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.808395 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.808400 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.808404 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.808408 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.808413 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.808417 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.808421 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.808432 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.808436 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.808441 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.808445 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.808450 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.808454 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.808458 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.808462 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.808467 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.808471 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.808475 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.808479 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:56:26.808484 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.808488 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.808492 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.808496 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.808501 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.808505 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.808509 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.808514 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.808518 | controller |     - '    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.808522 | controller |     - '    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile
2026-03-14 09:56:26.808526 | controller |       = /etc/pki/tls/certs/mtls-ca.crt'''
2026-03-14 09:56:26.808531 | controller |     - '    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.808535 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.808543 | controller |     - '    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.808547 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.808551 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.808556 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.808560 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.808564 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.808569 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.808573 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.808577 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.808581 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.808587 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.808591 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.808607 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.808612 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.808616 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.808621 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.808630 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.808635 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.808639 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.808644 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.808648 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.808653 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:56:26.808657 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.808661 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.808665 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.808670 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.808674 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.808678 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.808691 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.808697 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.808706 | controller |     - '    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.808712 | controller |     - '    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled
2026-03-14 09:56:26.808716 | controller |       = true'''
2026-03-14 09:56:26.808720 | controller |     - '    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.808724 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.808729 | controller |     - '    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.808733 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.808737 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.808741 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.808746 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.808750 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.808754 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.808758 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.808763 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.808767 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.808771 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.808775 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.808780 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.808784 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.808788 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.808792 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.808796 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.808801 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.808805 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.808809 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.808813 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.808818 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:56:26.808822 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.808833 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.808838 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.808842 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.808846 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.808850 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.808856 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.808861 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.808868 | controller |     - '    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.808872 | controller |     - '    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt'
2026-03-14 09:56:26.808876 | controller |     - '    logger.go:42: 09:50:03 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.808882 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.808886 | controller |     - '    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.808890 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.808895 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.808900 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.808905 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.808909 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.808913 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.808917 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.808921 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.808926 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.808930 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.808934 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.808938 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.808943 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.808947 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.808951 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.808955 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.808959 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.808964 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.808968 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.808972 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.808976 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:56:26.808980 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.808985 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.808989 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.808996 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.809000 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.809004 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.809008 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.809019 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.809024 | controller |     - '    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.809028 | controller |     - '    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key'
2026-03-14 09:56:26.809032 | controller |     - '    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.809037 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.809043 | controller |     - '    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.809047 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.809051 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.809056 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.809060 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.809065 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.809069 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.809073 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.809077 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.809082 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.809086 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.809090 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.809094 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.809098 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.809103 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.809107 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.809111 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.809115 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.809119 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.809124 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.809128 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.809132 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:56:26.809144 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.809149 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.809153 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.809157 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.809161 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.809166 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.809170 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.809174 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.809178 | controller |     - '    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.809183 | controller |     - '    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt'
2026-03-14 09:56:26.809187 | controller |     - '    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.809191 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.809195 | controller |     - '    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.809206 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.809210 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.809215 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.809219 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.809223 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.809227 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.809231 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.809236 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.809240 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.809244 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.809248 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.809253 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.809257 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.809261 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.809265 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.809270 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.809274 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.809278 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.809284 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.809288 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.809293 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:56:26.809297 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.809301 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.809305 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.809310 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.809314 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.809325 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.809329 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.809337 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.809341 | controller |     - '    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.809346 | controller |     - '    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.809350 | controller |       '']'''
2026-03-14 09:56:26.809354 | controller |     - '    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-decision-engine
2026-03-14 09:56:26.809358 | controller |       config...'''
2026-03-14 09:56:26.809363 | controller |     - '    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | Checking watcher-decision-engine
2026-03-14 09:56:26.809367 | controller |       config...'
2026-03-14 09:56:26.809371 | controller |     - '    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:56:26.809375 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.809380 | controller |     - '    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile
2026-03-14 09:56:26.809384 | controller |       = /etc/pki/tls/certs/mtls.crt '''
2026-03-14 09:56:26.809388 | controller |     - '    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | error: cannot exec into
2026-03-14 09:56:26.809392 | controller |       a container in a completed pod; current phase is Succeeded'
2026-03-14 09:56:26.809397 | controller |     - '    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | ++ echo'
2026-03-14 09:56:26.809401 | controller |     - '    logger.go:42: 09:50:04 | watcher-tls/2-patch-mtls | + ''['' 0 == 1 '']'''
2026-03-14 09:56:26.809405 | controller |     - '    logger.go:42: 09:50:05 | watcher-tls/2-patch-mtls | running command: [sh -c
2026-03-14 09:56:26.809416 | controller |       set -euxo pipefail'
2026-03-14 09:56:26.809420 | controller |     - '        '
2026-03-14 09:56:26.809425 | controller |     - '        oc project ${NAMESPACE}'
2026-03-14 09:56:26.809429 | controller |     - '        # Get pod names for each watcher service'
2026-03-14 09:56:26.809433 | controller |     - '        APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:56:26.809437 | controller |     - '        APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:56:26.809442 | controller |     - '        DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:56:26.809446 | controller |     - '        '
2026-03-14 09:56:26.809450 | controller |     - '        # Verify memcached mTLS config parameters in watcher-api config'
2026-03-14 09:56:26.809454 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.809459 | controller |     - '          echo "Checking watcher-api config..."'
2026-03-14 09:56:26.809463 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809476 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:56:26.809481 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809485 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.809489 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809495 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.809501 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809505 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:56:26.809510 | controller |     - '        '
2026-03-14 09:56:26.809514 | controller |     - '          # Verify mTLS config parameters in memcached backend config'
2026-03-14 09:56:26.809518 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809522 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:56:26.809526 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809531 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.809535 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809539 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.809543 | controller |     - '        else'
2026-03-14 09:56:26.809548 | controller |     - '          exit 1'
2026-03-14 09:56:26.809552 | controller |     - '        fi'
2026-03-14 09:56:26.809556 | controller |     - '        '
2026-03-14 09:56:26.809560 | controller |     - '        # Verify memcached mTLS config parameters in watcher-applier config'
2026-03-14 09:56:26.809565 | controller |     - '        if [ -n "${APPLIERPOD}" ]; then'
2026-03-14 09:56:26.809569 | controller |     - '          echo "Checking watcher-applier config..."'
2026-03-14 09:56:26.809573 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809577 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:56:26.809582 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809586 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.809592 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809609 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.809613 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809617 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:56:26.809622 | controller |     - '        '
2026-03-14 09:56:26.809626 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809630 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:56:26.809634 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809639 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.809643 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809647 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.809651 | controller |     - '        else'
2026-03-14 09:56:26.809656 | controller |     - '          exit 1'
2026-03-14 09:56:26.809660 | controller |     - '        fi'
2026-03-14 09:56:26.809666 | controller |     - '        '
2026-03-14 09:56:26.809670 | controller |     - '        # Verify memcached mTLS config parameters in watcher-decision-engine config'
2026-03-14 09:56:26.809675 | controller |     - '        if [ -n "${DECISIONENGINEPOD}" ]; then'
2026-03-14 09:56:26.809679 | controller |     - '          echo "Checking watcher-decision-engine config..."'
2026-03-14 09:56:26.809683 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809694 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:56:26.809700 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809705 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.809709 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809713 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.809717 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809722 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:56:26.809726 | controller |     - '        '
2026-03-14 09:56:26.809730 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809734 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:56:26.809739 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809743 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:56:26.809747 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.809751 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:56:26.809755 | controller |     - '        else'
2026-03-14 09:56:26.809760 | controller |     - '          exit 1'
2026-03-14 09:56:26.809764 | controller |     - '        fi'
2026-03-14 09:56:26.809768 | controller |     - '        ]'
2026-03-14 09:56:26.809772 | controller |     - '    logger.go:42: 09:50:05 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default'
2026-03-14 09:56:26.809777 | controller |     - '    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default"
2026-03-14 09:56:26.809781 | controller |       on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.809785 | controller |     - '    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api
2026-03-14 09:56:26.809789 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:56:26.809794 | controller |     - '    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.809798 | controller |     - '    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier
2026-03-14 09:56:26.809802 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:56:26.809806 | controller |     - '    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0'
2026-03-14 09:56:26.809811 | controller |     - '    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine
2026-03-14 09:56:26.809815 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:56:26.809819 | controller |     - '    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.809823 | controller |     - '    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0
2026-03-14 09:56:26.809828 | controller |       '']'''
2026-03-14 09:56:26.809832 | controller |     - '    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api
2026-03-14 09:56:26.809838 | controller |       config...'''
2026-03-14 09:56:26.809842 | controller |     - '    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | Checking watcher-api config...'
2026-03-14 09:56:26.809846 | controller |     - '    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.809851 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.809855 | controller |     - '    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile
2026-03-14 09:56:26.809859 | controller |       = /etc/pki/tls/certs/mtls.crt '''
2026-03-14 09:56:26.809863 | controller |     - '    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.809868 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.809872 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.809876 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.809880 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.809885 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.809889 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.809893 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.809898 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.809908 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.809912 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.809916 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.809921 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.809925 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.809929 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.809933 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.809938 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.809942 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.809946 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.809950 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.809954 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.809959 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.809963 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.809967 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.809971 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.809976 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.809980 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.809987 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.809991 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.809996 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.810000 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.810004 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.810009 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.810015 | controller |     - '    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.810019 | controller |     - '    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile
2026-03-14 09:56:26.810023 | controller |       = /etc/pki/tls/private/mtls.key'''
2026-03-14 09:56:26.810027 | controller |     - '    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.810032 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.810036 | controller |     - '    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.810040 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.810045 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.810049 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.810053 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.810058 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.810062 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.810066 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.810071 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.810075 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.810079 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.810083 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.810088 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.810099 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.810104 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.810109 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.810113 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.810117 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.810122 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.810126 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.810130 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.810136 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.810141 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.810145 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.810149 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.810154 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.810158 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.810162 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.810166 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.810171 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.810175 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.810179 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.810184 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.810188 | controller |     - '    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.810192 | controller |     - '    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile
2026-03-14 09:56:26.810196 | controller |       = /etc/pki/tls/certs/mtls-ca.crt'''
2026-03-14 09:56:26.810201 | controller |     - '    logger.go:42: 09:50:06 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.810205 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.810210 | controller |     - '    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.810214 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.810218 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.810223 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.810227 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.810231 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.810235 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.810240 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.810244 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.810248 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.810253 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.810257 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.810261 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.810266 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.810270 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.810274 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.810280 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.810284 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.810295 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.810299 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.810304 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.810308 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.810312 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.810317 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.810323 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.810327 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.810331 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.810336 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.810340 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.810344 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.810349 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.810353 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.810357 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.810362 | controller |     - '    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.810366 | controller |     - '    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled
2026-03-14 09:56:26.810370 | controller |       = true'''
2026-03-14 09:56:26.810374 | controller |     - '    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.810379 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.810383 | controller |     - '    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.810387 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.810392 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.810396 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.810400 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.810410 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.810415 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.810419 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.810427 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.810432 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.810438 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.810442 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.810447 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.810451 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.810455 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.810460 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.810464 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.810468 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.810472 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.810477 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.810481 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.810485 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.810497 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.810501 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.810506 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.810510 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.810514 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.810519 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.810523 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.810527 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.810533 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.810538 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.810542 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.810547 | controller |     - '    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.810551 | controller |     - '    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt'
2026-03-14 09:56:26.810555 | controller |     - '    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.810559 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.810564 | controller |     - '    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.810568 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.810572 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.810577 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.810581 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.810587 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.810591 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.810609 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.810613 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.810618 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.810622 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.810626 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.810632 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.810637 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.810641 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.810645 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.810650 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.810654 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.810658 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.810662 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.810667 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.810671 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.810675 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.810680 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.810684 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.810688 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.810699 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.810704 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.810708 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.810713 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.810717 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.810721 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.810726 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.810730 | controller |     - '    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.810734 | controller |     - '    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key'
2026-03-14 09:56:26.810738 | controller |     - '    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.810743 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.810747 | controller |     - '    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.810753 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.810758 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.810762 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.810766 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.810771 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.810775 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.810779 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.810785 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.810790 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.810794 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.810798 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.810803 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.810807 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.810811 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.810815 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.810820 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.810824 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.810828 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.810833 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.810837 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.810841 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.810845 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.810850 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.810854 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.810858 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.810862 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.810867 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.810871 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.810877 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.810881 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.810892 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.810898 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.810902 | controller |     - '    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.810907 | controller |     - '    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt'
2026-03-14 09:56:26.810911 | controller |     - '    logger.go:42: 09:50:07 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:56:26.810916 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.810920 | controller |     - '    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.810924 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.810929 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.810933 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.810937 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.810942 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.810946 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.810950 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.810956 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.810960 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.810965 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.810969 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.810973 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.810978 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.810982 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.810986 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.810991 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.811000 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.811005 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.811009 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.811018 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.811022 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.811027 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.811031 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.811035 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.811056 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.811063 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.811071 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.811076 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.811080 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.811085 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.811089 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.811093 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.811098 | controller |     - '    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.811108 | controller |     - '    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-applier-0
2026-03-14 09:56:26.811113 | controller |       '']'''
2026-03-14 09:56:26.811117 | controller |     - '    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-applier
2026-03-14 09:56:26.811122 | controller |       config...'''
2026-03-14 09:56:26.811126 | controller |     - '    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | Checking watcher-applier
2026-03-14 09:56:26.811130 | controller |       config...'
2026-03-14 09:56:26.811135 | controller |     - '    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile
2026-03-14 09:56:26.811139 | controller |       = /etc/pki/tls/certs/mtls.crt '''
2026-03-14 09:56:26.811146 | controller |     - '    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.811150 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.811155 | controller |     - '    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.811159 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.811163 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.811168 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.811172 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.811176 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.811181 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.811185 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.811189 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.811194 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.811198 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.811202 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.811206 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.811211 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.811215 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.811219 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.811225 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.811231 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.811235 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.811240 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.811244 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.811248 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:56:26.811252 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.811257 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.811261 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.811265 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.811269 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.811274 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.811278 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.811282 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.811287 | controller |     - '    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.811291 | controller |     - '    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile
2026-03-14 09:56:26.811295 | controller |       = /etc/pki/tls/private/mtls.key'''
2026-03-14 09:56:26.811299 | controller |     - '    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.811304 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.811313 | controller |     - '    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.811318 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.811322 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.811327 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.811331 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.811335 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.811340 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.811344 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.811348 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.811353 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.811357 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.811361 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.811365 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.811371 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.811376 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.811380 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.811384 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.811388 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.811393 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.811397 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.811401 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.811406 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:56:26.811410 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.811414 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.811418 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.811423 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.811427 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.811431 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.811437 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.811442 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.811451 | controller |     - '    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.811457 | controller |     - '    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile
2026-03-14 09:56:26.811462 | controller |       = /etc/pki/tls/certs/mtls-ca.crt'''
2026-03-14 09:56:26.811466 | controller |     - '    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.811470 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.811475 | controller |     - '    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.811479 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.811483 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.811488 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.811497 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.811502 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.811506 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.811511 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.811515 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.811521 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.811526 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.811531 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.811535 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.811539 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.811543 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.811548 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.811552 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.811556 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.811560 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.811564 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.811569 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.811573 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:56:26.811577 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.811581 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.811586 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.811590 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.811604 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.811612 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.811616 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.811620 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.811625 | controller |     - '    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.811629 | controller |     - '    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled
2026-03-14 09:56:26.811633 | controller |       = true'''
2026-03-14 09:56:26.811637 | controller |     - '    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.811642 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.811646 | controller |     - '    logger.go:42: 09:50:08 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.811650 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.811654 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.811659 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.811663 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.811667 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.811671 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.811677 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.811682 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.811686 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.811696 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.811700 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.811705 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.811709 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.811713 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.811717 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.811721 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.811726 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.811730 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.811734 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.811738 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.811743 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:56:26.811747 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.811751 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.811755 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.811760 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.811764 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.811768 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.811773 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.811778 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.811783 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.811787 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt'
2026-03-14 09:56:26.811791 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.811796 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.811800 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.811804 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.811808 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.811813 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.811819 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.811823 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.811827 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.811832 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.811836 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.811840 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.811844 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.811848 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.811853 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.811857 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.811861 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.811865 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.811870 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.811879 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.811884 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.811888 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.811892 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.811897 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:56:26.811901 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.811905 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.811909 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.811914 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.811918 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.811924 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.811928 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.811933 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.811937 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.811941 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key'
2026-03-14 09:56:26.811945 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.811950 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.811954 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.811958 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.811964 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.811968 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.811973 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.811977 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.811981 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.811985 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.811990 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.811994 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.811998 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.812002 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.812007 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.812011 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.812015 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.812019 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.812024 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.812028 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.812032 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.812036 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.812042 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.812047 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:56:26.812054 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.812063 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.812068 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.812072 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.812076 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.812081 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.812085 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.812089 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.812093 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.812098 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt'
2026-03-14 09:56:26.812102 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:56:26.812108 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.812112 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.812117 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.812121 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.812125 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.812129 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.812134 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.812138 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.812142 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.812146 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.812151 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.812155 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.812159 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.812163 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.812167 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.812172 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.812176 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.812180 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.812184 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.812188 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.812193 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.812197 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.812201 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:56:26.812205 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.812210 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.812214 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.812218 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.812222 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.812227 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.812231 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.812235 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.812248 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.812254 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-decision-engine-0
2026-03-14 09:56:26.812260 | controller |       '']'''
2026-03-14 09:56:26.812265 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-decision-engine
2026-03-14 09:56:26.812271 | controller |       config...'''
2026-03-14 09:56:26.812276 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | Checking watcher-decision-engine
2026-03-14 09:56:26.812281 | controller |       config...'
2026-03-14 09:56:26.812287 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile
2026-03-14 09:56:26.812292 | controller |       = /etc/pki/tls/certs/mtls.crt '''
2026-03-14 09:56:26.812297 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:56:26.812303 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.812309 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.812314 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.812320 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.812326 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.812332 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.812338 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.812343 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.812348 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.812354 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.812359 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.812365 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.812370 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.812378 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.812386 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.812391 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.812397 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.812403 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.812408 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.812414 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.812420 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.812426 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.812432 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.812438 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.812446 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.812452 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.812458 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.812464 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.812470 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.812476 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.812482 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.812487 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.812497 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.812513 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.812520 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile
2026-03-14 09:56:26.812527 | controller |       = /etc/pki/tls/private/mtls.key'''
2026-03-14 09:56:26.812532 | controller |     - '    logger.go:42: 09:50:09 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:56:26.812538 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.812544 | controller |     - '    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.812550 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.812556 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.812562 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.812568 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.812574 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.812579 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.812585 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.812590 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.812611 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.812621 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.812627 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.812633 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.812639 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.812644 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.812650 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.812656 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.812661 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.812669 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.812675 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.812681 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.812686 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.812692 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.812697 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.812703 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.812708 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.812714 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.812720 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.812729 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.812735 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.812741 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.812746 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.812752 | controller |     - '    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.812758 | controller |     - '    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile
2026-03-14 09:56:26.812764 | controller |       = /etc/pki/tls/certs/mtls-ca.crt'''
2026-03-14 09:56:26.812769 | controller |     - '    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:56:26.812775 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.812791 | controller |     - '    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.812797 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.812803 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.812809 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.812815 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.812820 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.812826 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.812832 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.812838 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.812844 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.812850 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.812855 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.812864 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.812869 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.812875 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.812880 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.812886 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.812892 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.812897 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.812903 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.812909 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.812915 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.812920 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.812926 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.812932 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.812937 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.812943 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.812948 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.812954 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.812960 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.812965 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.812971 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.812976 | controller |     - '    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.812982 | controller |     - '    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled
2026-03-14 09:56:26.812988 | controller |       = true'''
2026-03-14 09:56:26.812996 | controller |     - '    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:56:26.813002 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.813007 | controller |     - '    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.813013 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.813019 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.813033 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.813039 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.813045 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.813051 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.813059 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.813065 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.813070 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.813076 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.813082 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.813087 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.813093 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.813098 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.813106 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.813112 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.813118 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.813123 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.813129 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.813135 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.813141 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.813147 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.813153 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.813158 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.813164 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.813169 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.813175 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.813180 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.813186 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.813191 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.813197 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.813203 | controller |     - '    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.813209 | controller |     - '    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt'
2026-03-14 09:56:26.813214 | controller |     - '    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:56:26.813220 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.813225 | controller |     - '    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.813231 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.813237 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.813247 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.813253 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.813259 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.813265 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.813279 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.813285 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.813291 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.813296 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.813302 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.813307 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.813313 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.813319 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.813325 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.813330 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.813336 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.813342 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.813348 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.813354 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.813360 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.813366 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.813372 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.813378 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.813383 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.813389 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.813395 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.813401 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.813407 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.813412 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.813418 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.813429 | controller |     - '    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.813435 | controller |     - '    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key'
2026-03-14 09:56:26.813441 | controller |     - '    logger.go:42: 09:50:10 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:56:26.813450 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.813456 | controller |     - '    logger.go:42: 09:50:11 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.813461 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.813467 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.813473 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.813479 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.813485 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.813491 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.813497 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.813505 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.813512 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.813517 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.813523 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.813538 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.813545 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.813551 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.813557 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.813562 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.813567 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.813574 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.813580 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.813586 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.813609 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.813619 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.813625 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.813630 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.813636 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.813641 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.813646 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.813650 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.813654 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.813659 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.813666 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.813670 | controller |     - '    logger.go:42: 09:50:11 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.813674 | controller |     - '    logger.go:42: 09:50:11 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt'
2026-03-14 09:56:26.813678 | controller |     - '    logger.go:42: 09:50:11 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:56:26.813684 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.813689 | controller |     - '    logger.go:42: 09:50:11 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:56:26.813693 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.813697 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:56:26.813701 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_c17a:dcf25e84ff4f7e53556605c4d61756cb@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.813706 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.813710 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.813714 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.813718 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.813723 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.813727 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.813731 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.813736 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.813740 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.813744 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.813748 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.813753 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.813766 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.813771 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.813775 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.813779 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.813783 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.813788 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.813792 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.813796 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:56:26.813801 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:56:26.813807 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.813811 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:56:26.813816 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.813820 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.813824 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.813828 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.813833 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.813837 | controller |     - '    logger.go:42: 09:50:11 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.813841 | controller |     - '    logger.go:42: 09:50:11 | watcher-tls/2-patch-mtls | test step completed 2-patch-mtls'
2026-03-14 09:56:26.813845 | controller |     - '    logger.go:42: 09:50:11 | watcher-tls/3-disable-podlevel-tls | starting test
2026-03-14 09:56:26.813850 | controller |       step 3-disable-podlevel-tls'
2026-03-14 09:56:26.813854 | controller |     - '    logger.go:42: 09:50:11 | watcher-tls/3-disable-podlevel-tls | running command:
2026-03-14 09:56:26.813858 | controller |       [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op":
2026-03-14 09:56:26.813862 | controller |       "replace", "path": "/spec/apiServiceTemplate/tls/api", "value":{ "internal": {},
2026-03-14 09:56:26.813867 | controller |       "public": {} }}]'''
2026-03-14 09:56:26.813871 | controller |     - '        ]'
2026-03-14 09:56:26.813875 | controller |     - '    logger.go:42: 09:50:11 | watcher-tls/3-disable-podlevel-tls | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:56:26.813879 | controller |       patched'
2026-03-14 09:56:26.813884 | controller |     - '    logger.go:42: 09:50:11 | watcher-tls/3-disable-podlevel-tls | running command:
2026-03-14 09:56:26.813888 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.813892 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:56:26.813896 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.813901 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.813905 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.813909 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.813913 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.813918 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.813922 | controller |       ]'
2026-03-14 09:56:26.813926 | controller |     - '        # check that watcher internal endpoint does not use https'
2026-03-14 09:56:26.813930 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.813935 | controller |       infra-optim | grep internal  | [ $(grep -c https) == 0 ]'
2026-03-14 09:56:26.813939 | controller |     - '        # check that watcher public endpoint does use https'
2026-03-14 09:56:26.813943 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.813948 | controller |       infra-optim | grep public  | [ $(grep -c https) == 1 ]'
2026-03-14 09:56:26.813952 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.813956 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.813960 | controller |       == "" ]; then'
2026-03-14 09:56:26.813965 | controller |     - '            exit 0'
2026-03-14 09:56:26.813969 | controller |     - '        fi'
2026-03-14 09:56:26.813973 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.813977 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.813985 | controller |     - '        counter=0'
2026-03-14 09:56:26.813989 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.813995 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.814000 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.814004 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.814008 | controller |     - '            fi'
2026-03-14 09:56:26.814012 | controller |     - '        done'
2026-03-14 09:56:26.814017 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.814027 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.814031 | controller |     - '          exit 1'
2026-03-14 09:56:26.814036 | controller |     - '        else'
2026-03-14 09:56:26.814040 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.814044 | controller |     - '        fi'
2026-03-14 09:56:26.814048 | controller |     - '        ]'
2026-03-14 09:56:26.814053 | controller |     - '    logger.go:42: 09:50:11 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:56:26.814057 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.814061 | controller |       -c Type'
2026-03-14 09:56:26.814065 | controller |     - '    logger.go:42: 09:50:11 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.814070 | controller |     - '    logger.go:42: 09:50:13 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:56:26.814074 | controller |       1 '']'''
2026-03-14 09:56:26.814078 | controller |     - '    logger.go:42: 09:50:13 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher'
2026-03-14 09:56:26.814082 | controller |     - '    logger.go:42: 09:50:13 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n
2026-03-14 09:56:26.814087 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.814091 | controller |       -c Type -c ID'
2026-03-14 09:56:26.814095 | controller |     - '    logger.go:42: 09:50:13 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print
2026-03-14 09:56:26.814099 | controller |       $1}'''
2026-03-14 09:56:26.814103 | controller |     - '    logger.go:42: 09:50:16 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa'
2026-03-14 09:56:26.814108 | controller |     - '    logger.go:42: 09:50:16 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:56:26.814112 | controller |       watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.814116 | controller |     - '    logger.go:42: 09:50:16 | watcher-tls/3-disable-podlevel-tls | + ''['' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.814120 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa '']'''
2026-03-14 09:56:26.814125 | controller |     - '    logger.go:42: 09:50:16 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:56:26.814129 | controller |       watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.814133 | controller |     - '    logger.go:42: 09:50:16 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.814138 | controller |       '']'''
2026-03-14 09:56:26.814143 | controller |     - '    logger.go:42: 09:50:16 | watcher-tls/3-disable-podlevel-tls | + grep internal'
2026-03-14 09:56:26.814148 | controller |     - '    logger.go:42: 09:50:16 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim'
2026-03-14 09:56:26.814152 | controller |     - '    logger.go:42: 09:50:16 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https'
2026-03-14 09:56:26.814156 | controller |     - '    logger.go:42: 09:50:16 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:56:26.814160 | controller |       watcher-kuttl-default openstackclient -- openstack endpoint list'
2026-03-14 09:56:26.814165 | controller |     - '    logger.go:42: 09:50:18 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:56:26.814169 | controller |       0 '']'''
2026-03-14 09:56:26.814173 | controller |     - '    logger.go:42: 09:50:19 | watcher-tls/3-disable-podlevel-tls | running command:
2026-03-14 09:56:26.814177 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.814191 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:56:26.814195 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.814200 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.814204 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.814208 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.814212 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.814218 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.814223 | controller |       ]'
2026-03-14 09:56:26.814227 | controller |     - '        # check that watcher internal endpoint does not use https'
2026-03-14 09:56:26.814231 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.814235 | controller |       infra-optim | grep internal  | [ $(grep -c https) == 0 ]'
2026-03-14 09:56:26.814240 | controller |     - '        # check that watcher public endpoint does use https'
2026-03-14 09:56:26.814244 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.814248 | controller |       infra-optim | grep public  | [ $(grep -c https) == 1 ]'
2026-03-14 09:56:26.814252 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.814257 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.814261 | controller |       == "" ]; then'
2026-03-14 09:56:26.814265 | controller |     - '            exit 0'
2026-03-14 09:56:26.814269 | controller |     - '        fi'
2026-03-14 09:56:26.814274 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.814278 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.814282 | controller |     - '        counter=0'
2026-03-14 09:56:26.814286 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.814290 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.814295 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.814305 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.814309 | controller |     - '            fi'
2026-03-14 09:56:26.814313 | controller |     - '        done'
2026-03-14 09:56:26.814318 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.814322 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.814326 | controller |     - '          exit 1'
2026-03-14 09:56:26.814331 | controller |     - '        else'
2026-03-14 09:56:26.814335 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.814340 | controller |     - '        fi'
2026-03-14 09:56:26.814347 | controller |     - '        ]'
2026-03-14 09:56:26.814352 | controller |     - '    logger.go:42: 09:50:19 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:56:26.814356 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.814362 | controller |       -c Type'
2026-03-14 09:56:26.814367 | controller |     - '    logger.go:42: 09:50:19 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.814372 | controller |     - '    logger.go:42: 09:50:21 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:56:26.814376 | controller |       1 '']'''
2026-03-14 09:56:26.814380 | controller |     - '    logger.go:42: 09:50:21 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher'
2026-03-14 09:56:26.814384 | controller |     - '    logger.go:42: 09:50:21 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print
2026-03-14 09:56:26.814388 | controller |       $1}'''
2026-03-14 09:56:26.814393 | controller |     - '    logger.go:42: 09:50:21 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n
2026-03-14 09:56:26.814397 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.814403 | controller |       -c Type -c ID'
2026-03-14 09:56:26.814407 | controller |     - '    logger.go:42: 09:50:23 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa'
2026-03-14 09:56:26.814412 | controller |     - '    logger.go:42: 09:50:23 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:56:26.814416 | controller |       watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.814420 | controller |     - '    logger.go:42: 09:50:24 | watcher-tls/3-disable-podlevel-tls | + ''['' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.814424 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa '']'''
2026-03-14 09:56:26.814429 | controller |     - '    logger.go:42: 09:50:24 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:56:26.814433 | controller |       watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.814437 | controller |     - '    logger.go:42: 09:50:24 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.814441 | controller |       '']'''
2026-03-14 09:56:26.814445 | controller |     - '    logger.go:42: 09:50:24 | watcher-tls/3-disable-podlevel-tls | + grep internal'
2026-03-14 09:56:26.814450 | controller |     - '    logger.go:42: 09:50:24 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim'
2026-03-14 09:56:26.814454 | controller |     - '    logger.go:42: 09:50:24 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https'
2026-03-14 09:56:26.814458 | controller |     - '    logger.go:42: 09:50:24 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:56:26.814462 | controller |       watcher-kuttl-default openstackclient -- openstack endpoint list'
2026-03-14 09:56:26.814467 | controller |     - '    logger.go:42: 09:50:26 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:56:26.814471 | controller |       0 '']'''
2026-03-14 09:56:26.814475 | controller |     - '    logger.go:42: 09:50:27 | watcher-tls/3-disable-podlevel-tls | running command:
2026-03-14 09:56:26.814479 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.814483 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:56:26.814488 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.814492 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.814496 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.814500 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.814504 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.814509 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.814513 | controller |       ]'
2026-03-14 09:56:26.814520 | controller |     - '        # check that watcher internal endpoint does not use https'
2026-03-14 09:56:26.814524 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.814528 | controller |       infra-optim | grep internal  | [ $(grep -c https) == 0 ]'
2026-03-14 09:56:26.814535 | controller |     - '        # check that watcher public endpoint does use https'
2026-03-14 09:56:26.814540 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.814544 | controller |       infra-optim | grep public  | [ $(grep -c https) == 1 ]'
2026-03-14 09:56:26.814548 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.814552 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.814557 | controller |       == "" ]; then'
2026-03-14 09:56:26.814561 | controller |     - '            exit 0'
2026-03-14 09:56:26.814565 | controller |     - '        fi'
2026-03-14 09:56:26.814569 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.814574 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.814580 | controller |     - '        counter=0'
2026-03-14 09:56:26.814614 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.814627 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.814632 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.814636 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.814641 | controller |     - '            fi'
2026-03-14 09:56:26.814645 | controller |     - '        done'
2026-03-14 09:56:26.814650 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.814654 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.814658 | controller |     - '          exit 1'
2026-03-14 09:56:26.814663 | controller |     - '        else'
2026-03-14 09:56:26.814667 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.814671 | controller |     - '        fi'
2026-03-14 09:56:26.814676 | controller |     - '        ]'
2026-03-14 09:56:26.814680 | controller |     - '    logger.go:42: 09:50:27 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:56:26.814684 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.814688 | controller |       -c Type'
2026-03-14 09:56:26.814693 | controller |     - '    logger.go:42: 09:50:27 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.814697 | controller |     - '    logger.go:42: 09:50:29 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:56:26.814702 | controller |       1 '']'''
2026-03-14 09:56:26.814706 | controller |     - '    logger.go:42: 09:50:29 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher'
2026-03-14 09:56:26.814710 | controller |     - '    logger.go:42: 09:50:29 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print
2026-03-14 09:56:26.814714 | controller |       $1}'''
2026-03-14 09:56:26.814719 | controller |     - '    logger.go:42: 09:50:29 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n
2026-03-14 09:56:26.814723 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.814727 | controller |       -c Type -c ID'
2026-03-14 09:56:26.814732 | controller |     - '    logger.go:42: 09:50:31 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa'
2026-03-14 09:56:26.814736 | controller |     - '    logger.go:42: 09:50:31 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:56:26.814740 | controller |       watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.814745 | controller |     - '    logger.go:42: 09:50:31 | watcher-tls/3-disable-podlevel-tls | + ''['' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.814749 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa '']'''
2026-03-14 09:56:26.814753 | controller |     - '    logger.go:42: 09:50:31 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:56:26.814757 | controller |       watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.814762 | controller |     - '    logger.go:42: 09:50:31 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.814766 | controller |       '']'''
2026-03-14 09:56:26.814773 | controller |     - '    logger.go:42: 09:50:31 | watcher-tls/3-disable-podlevel-tls | + grep internal'
2026-03-14 09:56:26.814777 | controller |     - '    logger.go:42: 09:50:31 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim'
2026-03-14 09:56:26.814781 | controller |     - '    logger.go:42: 09:50:31 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:56:26.814785 | controller |       watcher-kuttl-default openstackclient -- openstack endpoint list'
2026-03-14 09:56:26.814790 | controller |     - '    logger.go:42: 09:50:31 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https'
2026-03-14 09:56:26.814794 | controller |     - '    logger.go:42: 09:50:33 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:56:26.814798 | controller |       0 '']'''
2026-03-14 09:56:26.814803 | controller |     - '    logger.go:42: 09:50:34 | watcher-tls/3-disable-podlevel-tls | running command:
2026-03-14 09:56:26.814807 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.814813 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:56:26.814818 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.814822 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.814826 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.814830 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.814835 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.814839 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.814843 | controller |       ]'
2026-03-14 09:56:26.814848 | controller |     - '        # check that watcher internal endpoint does not use https'
2026-03-14 09:56:26.814852 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.814856 | controller |       infra-optim | grep internal  | [ $(grep -c https) == 0 ]'
2026-03-14 09:56:26.814860 | controller |     - '        # check that watcher public endpoint does use https'
2026-03-14 09:56:26.814865 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.814869 | controller |       infra-optim | grep public  | [ $(grep -c https) == 1 ]'
2026-03-14 09:56:26.814873 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.814878 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.814882 | controller |       == "" ]; then'
2026-03-14 09:56:26.814886 | controller |     - '            exit 0'
2026-03-14 09:56:26.814890 | controller |     - '        fi'
2026-03-14 09:56:26.814895 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.814905 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.814912 | controller |     - '        counter=0'
2026-03-14 09:56:26.814916 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.814921 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.814925 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.814929 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.814934 | controller |     - '            fi'
2026-03-14 09:56:26.814938 | controller |     - '        done'
2026-03-14 09:56:26.814942 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.814947 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.814951 | controller |     - '          exit 1'
2026-03-14 09:56:26.814955 | controller |     - '        else'
2026-03-14 09:56:26.814960 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.814964 | controller |     - '        fi'
2026-03-14 09:56:26.814968 | controller |     - '        ]'
2026-03-14 09:56:26.814972 | controller |     - '    logger.go:42: 09:50:34 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:56:26.814977 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.814981 | controller |       -c Type'
2026-03-14 09:56:26.814985 | controller |     - '    logger.go:42: 09:50:34 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.814990 | controller |     - '    logger.go:42: 09:50:36 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:56:26.814994 | controller |       1 '']'''
2026-03-14 09:56:26.814998 | controller |     - '    logger.go:42: 09:50:36 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher'
2026-03-14 09:56:26.815003 | controller |     - '    logger.go:42: 09:50:36 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n
2026-03-14 09:56:26.815007 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.815011 | controller |       -c Type -c ID'
2026-03-14 09:56:26.815015 | controller |     - '    logger.go:42: 09:50:36 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print
2026-03-14 09:56:26.815020 | controller |       $1}'''
2026-03-14 09:56:26.815026 | controller |     - '    logger.go:42: 09:50:38 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa'
2026-03-14 09:56:26.815032 | controller |     - '    logger.go:42: 09:50:38 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:56:26.815036 | controller |       watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.815041 | controller |     - '    logger.go:42: 09:50:39 | watcher-tls/3-disable-podlevel-tls | + ''['' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.815062 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa '']'''
2026-03-14 09:56:26.815068 | controller |     - '    logger.go:42: 09:50:39 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:56:26.815072 | controller |       watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.815077 | controller |     - '    logger.go:42: 09:50:39 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.815081 | controller |       '']'''
2026-03-14 09:56:26.815086 | controller |     - '    logger.go:42: 09:50:39 | watcher-tls/3-disable-podlevel-tls | + grep internal'
2026-03-14 09:56:26.815090 | controller |     - '    logger.go:42: 09:50:39 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim'
2026-03-14 09:56:26.815094 | controller |     - '    logger.go:42: 09:50:39 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https'
2026-03-14 09:56:26.815098 | controller |     - '    logger.go:42: 09:50:39 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:56:26.815103 | controller |       watcher-kuttl-default openstackclient -- openstack endpoint list'
2026-03-14 09:56:26.815107 | controller |     - '    logger.go:42: 09:50:41 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:56:26.815111 | controller |       0 '']'''
2026-03-14 09:56:26.815116 | controller |     - '    logger.go:42: 09:50:42 | watcher-tls/3-disable-podlevel-tls | running command:
2026-03-14 09:56:26.815120 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.815124 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:56:26.815129 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.815133 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.815137 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.815141 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.815146 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.815150 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.815154 | controller |       ]'
2026-03-14 09:56:26.815159 | controller |     - '        # check that watcher internal endpoint does not use https'
2026-03-14 09:56:26.815163 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.815170 | controller |       infra-optim | grep internal  | [ $(grep -c https) == 0 ]'
2026-03-14 09:56:26.815174 | controller |     - '        # check that watcher public endpoint does use https'
2026-03-14 09:56:26.815178 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.815182 | controller |       infra-optim | grep public  | [ $(grep -c https) == 1 ]'
2026-03-14 09:56:26.815187 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.815192 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.815197 | controller |       == "" ]; then'
2026-03-14 09:56:26.815212 | controller |     - '            exit 0'
2026-03-14 09:56:26.815219 | controller |     - '        fi'
2026-03-14 09:56:26.815225 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.815231 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.815237 | controller |     - '        counter=0'
2026-03-14 09:56:26.815245 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.815250 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.815254 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.815258 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.815263 | controller |     - '            fi'
2026-03-14 09:56:26.815267 | controller |     - '        done'
2026-03-14 09:56:26.815271 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.815276 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.815280 | controller |     - '          exit 1'
2026-03-14 09:56:26.815284 | controller |     - '        else'
2026-03-14 09:56:26.815288 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.815293 | controller |     - '        fi'
2026-03-14 09:56:26.815297 | controller |     - '        ]'
2026-03-14 09:56:26.815303 | controller |     - '    logger.go:42: 09:50:42 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:56:26.815309 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.815315 | controller |       -c Type'
2026-03-14 09:56:26.815321 | controller |     - '    logger.go:42: 09:50:42 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.815327 | controller |     - '    logger.go:42: 09:50:44 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:56:26.815332 | controller |       1 '']'''
2026-03-14 09:56:26.815336 | controller |     - '    logger.go:42: 09:50:44 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n
2026-03-14 09:56:26.815340 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.815345 | controller |       -c Type -c ID'
2026-03-14 09:56:26.815349 | controller |     - '    logger.go:42: 09:50:44 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print
2026-03-14 09:56:26.815353 | controller |       $1}'''
2026-03-14 09:56:26.815357 | controller |     - '    logger.go:42: 09:50:44 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher'
2026-03-14 09:56:26.815362 | controller |     - '    logger.go:42: 09:50:46 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa'
2026-03-14 09:56:26.815366 | controller |     - '    logger.go:42: 09:50:46 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:56:26.815370 | controller |       watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.815374 | controller |     - '    logger.go:42: 09:50:46 | watcher-tls/3-disable-podlevel-tls | + ''['' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.815379 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa '']'''
2026-03-14 09:56:26.815383 | controller |     - '    logger.go:42: 09:50:46 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:56:26.815387 | controller |       watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.815391 | controller |     - '    logger.go:42: 09:50:47 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.815395 | controller |       '']'''
2026-03-14 09:56:26.815400 | controller |     - '    logger.go:42: 09:50:47 | watcher-tls/3-disable-podlevel-tls | + grep internal'
2026-03-14 09:56:26.815404 | controller |     - '    logger.go:42: 09:50:47 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim'
2026-03-14 09:56:26.815408 | controller |     - '    logger.go:42: 09:50:47 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:56:26.815412 | controller |       watcher-kuttl-default openstackclient -- openstack endpoint list'
2026-03-14 09:56:26.815416 | controller |     - '    logger.go:42: 09:50:47 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https'
2026-03-14 09:56:26.815421 | controller |     - '    logger.go:42: 09:50:49 | watcher-tls/3-disable-podlevel-tls | + ''['' 0 ==
2026-03-14 09:56:26.815425 | controller |       0 '']'''
2026-03-14 09:56:26.815431 | controller |     - '    logger.go:42: 09:50:49 | watcher-tls/3-disable-podlevel-tls | + grep public'
2026-03-14 09:56:26.815436 | controller |     - '    logger.go:42: 09:50:49 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim'
2026-03-14 09:56:26.815442 | controller |     - '    logger.go:42: 09:50:49 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:56:26.815446 | controller |       watcher-kuttl-default openstackclient -- openstack endpoint list'
2026-03-14 09:56:26.815450 | controller |     - '    logger.go:42: 09:50:49 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https'
2026-03-14 09:56:26.815454 | controller |     - '    logger.go:42: 09:50:51 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:56:26.815459 | controller |       1 '']'''
2026-03-14 09:56:26.815463 | controller |     - '    logger.go:42: 09:50:51 | watcher-tls/3-disable-podlevel-tls | ++ oc get pods
2026-03-14 09:56:26.815467 | controller |       -n openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:56:26.815471 | controller |     - '    logger.go:42: 09:50:51 | watcher-tls/3-disable-podlevel-tls | + ''['' ''''
2026-03-14 09:56:26.815476 | controller |       == '''' '']'''
2026-03-14 09:56:26.815480 | controller |     - '    logger.go:42: 09:50:51 | watcher-tls/3-disable-podlevel-tls | + exit 0'
2026-03-14 09:56:26.815484 | controller |     - '    logger.go:42: 09:50:52 | watcher-tls/3-disable-podlevel-tls | running command:
2026-03-14 09:56:26.815488 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.815493 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:56:26.815497 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.815509 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.815514 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.815518 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.815522 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.815526 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.815531 | controller |       ]'
2026-03-14 09:56:26.815535 | controller |     - '        # check that watcher internal endpoint does not use https'
2026-03-14 09:56:26.815539 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.815543 | controller |       infra-optim | grep internal  | [ $(grep -c https) == 0 ]'
2026-03-14 09:56:26.815548 | controller |     - '        # check that watcher public endpoint does use https'
2026-03-14 09:56:26.815552 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.815556 | controller |       infra-optim | grep public  | [ $(grep -c https) == 1 ]'
2026-03-14 09:56:26.815560 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.815564 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.815569 | controller |       == "" ]; then'
2026-03-14 09:56:26.815573 | controller |     - '            exit 0'
2026-03-14 09:56:26.815577 | controller |     - '        fi'
2026-03-14 09:56:26.815581 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.815585 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.815590 | controller |     - '        counter=0'
2026-03-14 09:56:26.815619 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.815627 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.815631 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.815635 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.815639 | controller |     - '            fi'
2026-03-14 09:56:26.815644 | controller |     - '        done'
2026-03-14 09:56:26.815648 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.815652 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.815656 | controller |     - '          exit 1'
2026-03-14 09:56:26.815661 | controller |     - '        else'
2026-03-14 09:56:26.815665 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.815669 | controller |     - '        fi'
2026-03-14 09:56:26.815676 | controller |     - '        ]'
2026-03-14 09:56:26.815680 | controller |     - '    logger.go:42: 09:50:52 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:56:26.815684 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.815688 | controller |       -c Type'
2026-03-14 09:56:26.815692 | controller |     - '    logger.go:42: 09:50:52 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.815697 | controller |     - '    logger.go:42: 09:50:54 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:56:26.815701 | controller |       1 '']'''
2026-03-14 09:56:26.815705 | controller |     - '    logger.go:42: 09:50:54 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher'
2026-03-14 09:56:26.815709 | controller |     - '    logger.go:42: 09:50:54 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print
2026-03-14 09:56:26.815713 | controller |       $1}'''
2026-03-14 09:56:26.815718 | controller |     - '    logger.go:42: 09:50:54 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n
2026-03-14 09:56:26.815722 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.815726 | controller |       -c Type -c ID'
2026-03-14 09:56:26.815730 | controller |     - '    logger.go:42: 09:50:56 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa'
2026-03-14 09:56:26.815734 | controller |     - '    logger.go:42: 09:50:56 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:56:26.815739 | controller |       watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.815743 | controller |     - '    logger.go:42: 09:50:56 | watcher-tls/3-disable-podlevel-tls | + ''['' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.815747 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa '']'''
2026-03-14 09:56:26.815752 | controller |     - '    logger.go:42: 09:50:56 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:56:26.815758 | controller |       watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.815762 | controller |     - '    logger.go:42: 09:50:57 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.815767 | controller |       '']'''
2026-03-14 09:56:26.815771 | controller |     - '    logger.go:42: 09:50:57 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:56:26.815775 | controller |       watcher-kuttl-default openstackclient -- openstack endpoint list'
2026-03-14 09:56:26.815779 | controller |     - '    logger.go:42: 09:50:57 | watcher-tls/3-disable-podlevel-tls | + grep internal'
2026-03-14 09:56:26.815783 | controller |     - '    logger.go:42: 09:50:57 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https'
2026-03-14 09:56:26.815788 | controller |     - '    logger.go:42: 09:50:57 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim'
2026-03-14 09:56:26.815792 | controller |     - '    logger.go:42: 09:50:59 | watcher-tls/3-disable-podlevel-tls | + ''['' 0 ==
2026-03-14 09:56:26.815796 | controller |       0 '']'''
2026-03-14 09:56:26.815800 | controller |     - '    logger.go:42: 09:50:59 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:56:26.815804 | controller |       watcher-kuttl-default openstackclient -- openstack endpoint list'
2026-03-14 09:56:26.815816 | controller |     - '    logger.go:42: 09:50:59 | watcher-tls/3-disable-podlevel-tls | + grep public'
2026-03-14 09:56:26.815822 | controller |     - '    logger.go:42: 09:50:59 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https'
2026-03-14 09:56:26.815827 | controller |     - '    logger.go:42: 09:50:59 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim'
2026-03-14 09:56:26.815831 | controller |     - '    logger.go:42: 09:51:01 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:56:26.815835 | controller |       1 '']'''
2026-03-14 09:56:26.815839 | controller |     - '    logger.go:42: 09:51:01 | watcher-tls/3-disable-podlevel-tls | ++ oc get pods
2026-03-14 09:56:26.815844 | controller |       -n openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:56:26.815848 | controller |     - '    logger.go:42: 09:51:01 | watcher-tls/3-disable-podlevel-tls | + ''['' ''''
2026-03-14 09:56:26.815854 | controller |       == '''' '']'''
2026-03-14 09:56:26.815858 | controller |     - '    logger.go:42: 09:51:01 | watcher-tls/3-disable-podlevel-tls | + exit 0'
2026-03-14 09:56:26.815862 | controller |     - '    logger.go:42: 09:51:01 | watcher-tls/3-disable-podlevel-tls | test step completed
2026-03-14 09:56:26.815867 | controller |       3-disable-podlevel-tls'
2026-03-14 09:56:26.815871 | controller |     - '    logger.go:42: 09:51:01 | watcher-tls/4-deploy-without-route | starting test
2026-03-14 09:56:26.815875 | controller |       step 4-deploy-without-route'
2026-03-14 09:56:26.815880 | controller |     - '    logger.go:42: 09:51:01 | watcher-tls/4-deploy-without-route | running command:
2026-03-14 09:56:26.815884 | controller |       [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op":
2026-03-14 09:56:26.815888 | controller |       "replace", "path": "/spec/apiServiceTemplate/override", "value":{"service": { "internal":
2026-03-14 09:56:26.815892 | controller |       {}, "public": { "metadata": { "annotations": { "metallb.universe.tf/address-pool":
2026-03-14 09:56:26.815896 | controller |       "ctlplane", "metallb.universe.tf/allow-shared-ip": "ctlplane" } }, "spec": { "type":
2026-03-14 09:56:26.815901 | controller |       "LoadBalancer" } } } }}]'''
2026-03-14 09:56:26.815905 | controller |     - '        ]'
2026-03-14 09:56:26.815909 | controller |     - '    logger.go:42: 09:51:01 | watcher-tls/4-deploy-without-route | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:56:26.815913 | controller |       patched'
2026-03-14 09:56:26.815918 | controller |     - '    logger.go:42: 09:51:01 | watcher-tls/4-deploy-without-route | running command:
2026-03-14 09:56:26.815922 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.815926 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:56:26.815930 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.815935 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.815939 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.815943 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.815947 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.815952 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.815956 | controller |       ]'
2026-03-14 09:56:26.815960 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.815964 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.815969 | controller |       == "" ]; then'
2026-03-14 09:56:26.815973 | controller |     - '            exit 0'
2026-03-14 09:56:26.815979 | controller |     - '        fi'
2026-03-14 09:56:26.815983 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.815988 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.815992 | controller |     - '        counter=0'
2026-03-14 09:56:26.815996 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.816000 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.816005 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.816009 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.816013 | controller |     - '            fi'
2026-03-14 09:56:26.816017 | controller |     - '        done'
2026-03-14 09:56:26.816022 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.816026 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.816030 | controller |     - '          exit 1'
2026-03-14 09:56:26.816034 | controller |     - '        else'
2026-03-14 09:56:26.816042 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.816046 | controller |     - '        fi'
2026-03-14 09:56:26.816050 | controller |     - '        ]'
2026-03-14 09:56:26.816055 | controller |     - '    logger.go:42: 09:51:01 | watcher-tls/4-deploy-without-route | + oc exec -n
2026-03-14 09:56:26.816060 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.816065 | controller |       -c Type'
2026-03-14 09:56:26.816069 | controller |     - '    logger.go:42: 09:51:01 | watcher-tls/4-deploy-without-route | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.816073 | controller |     - '    logger.go:42: 09:51:03 | watcher-tls/4-deploy-without-route | + ''['' 1 ==
2026-03-14 09:56:26.816077 | controller |       1 '']'''
2026-03-14 09:56:26.816082 | controller |     - '    logger.go:42: 09:51:03 | watcher-tls/4-deploy-without-route | ++ oc exec -n
2026-03-14 09:56:26.816086 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.816090 | controller |       -c Type -c ID'
2026-03-14 09:56:26.816095 | controller |     - '    logger.go:42: 09:51:03 | watcher-tls/4-deploy-without-route | ++ awk ''{print
2026-03-14 09:56:26.816099 | controller |       $1}'''
2026-03-14 09:56:26.816103 | controller |     - '    logger.go:42: 09:51:03 | watcher-tls/4-deploy-without-route | ++ grep watcher'
2026-03-14 09:56:26.816114 | controller |     - '    logger.go:42: 09:51:05 | watcher-tls/4-deploy-without-route | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa'
2026-03-14 09:56:26.816118 | controller |     - '    logger.go:42: 09:51:05 | watcher-tls/4-deploy-without-route | ++ oc get -n
2026-03-14 09:56:26.816122 | controller |       watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.816127 | controller |     - '    logger.go:42: 09:51:06 | watcher-tls/4-deploy-without-route | + ''['' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.816131 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa '']'''
2026-03-14 09:56:26.816135 | controller |     - '    logger.go:42: 09:51:06 | watcher-tls/4-deploy-without-route | ++ oc get -n
2026-03-14 09:56:26.816139 | controller |       watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.816144 | controller |     - '    logger.go:42: 09:51:06 | watcher-tls/4-deploy-without-route | + ''['' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.816148 | controller |       '']'''
2026-03-14 09:56:26.816152 | controller |     - '    logger.go:42: 09:51:06 | watcher-tls/4-deploy-without-route | ++ oc get pods
2026-03-14 09:56:26.816157 | controller |       -n openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:56:26.816161 | controller |     - '    logger.go:42: 09:51:06 | watcher-tls/4-deploy-without-route | + ''['' ''''
2026-03-14 09:56:26.816165 | controller |       == '''' '']'''
2026-03-14 09:56:26.816169 | controller |     - '    logger.go:42: 09:51:06 | watcher-tls/4-deploy-without-route | + exit 0'
2026-03-14 09:56:26.816176 | controller |     - '    logger.go:42: 09:51:07 | watcher-tls/4-deploy-without-route | running command:
2026-03-14 09:56:26.816180 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.816184 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:56:26.816189 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.816193 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.816197 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.816201 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.816205 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.816210 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.816214 | controller |       ]'
2026-03-14 09:56:26.816218 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.816222 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.816227 | controller |       == "" ]; then'
2026-03-14 09:56:26.816231 | controller |     - '            exit 0'
2026-03-14 09:56:26.816235 | controller |     - '        fi'
2026-03-14 09:56:26.816239 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.816243 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.816250 | controller |     - '        counter=0'
2026-03-14 09:56:26.816254 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.816258 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.816262 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.816266 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.816271 | controller |     - '            fi'
2026-03-14 09:56:26.816275 | controller |     - '        done'
2026-03-14 09:56:26.816279 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.816283 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.816288 | controller |     - '          exit 1'
2026-03-14 09:56:26.816292 | controller |     - '        else'
2026-03-14 09:56:26.816296 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.816300 | controller |     - '        fi'
2026-03-14 09:56:26.816304 | controller |     - '        ]'
2026-03-14 09:56:26.816309 | controller |     - '    logger.go:42: 09:51:07 | watcher-tls/4-deploy-without-route | + oc exec -n
2026-03-14 09:56:26.816313 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.816317 | controller |       -c Type'
2026-03-14 09:56:26.816321 | controller |     - '    logger.go:42: 09:51:07 | watcher-tls/4-deploy-without-route | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.816326 | controller |     - '    logger.go:42: 09:51:09 | watcher-tls/4-deploy-without-route | + ''['' 1 ==
2026-03-14 09:56:26.816330 | controller |       1 '']'''
2026-03-14 09:56:26.816334 | controller |     - '    logger.go:42: 09:51:09 | watcher-tls/4-deploy-without-route | ++ grep watcher'
2026-03-14 09:56:26.816338 | controller |     - '    logger.go:42: 09:51:09 | watcher-tls/4-deploy-without-route | ++ awk ''{print
2026-03-14 09:56:26.816342 | controller |       $1}'''
2026-03-14 09:56:26.816347 | controller |     - '    logger.go:42: 09:51:09 | watcher-tls/4-deploy-without-route | ++ oc exec -n
2026-03-14 09:56:26.816351 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.816355 | controller |       -c Type -c ID'
2026-03-14 09:56:26.816359 | controller |     - '    logger.go:42: 09:51:11 | watcher-tls/4-deploy-without-route | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa'
2026-03-14 09:56:26.816363 | controller |     - '    logger.go:42: 09:51:11 | watcher-tls/4-deploy-without-route | ++ oc get -n
2026-03-14 09:56:26.816368 | controller |       watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.816372 | controller |     - '    logger.go:42: 09:51:11 | watcher-tls/4-deploy-without-route | + ''['' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.816376 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa '']'''
2026-03-14 09:56:26.816380 | controller |     - '    logger.go:42: 09:51:11 | watcher-tls/4-deploy-without-route | ++ oc get -n
2026-03-14 09:56:26.816390 | controller |       watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.816395 | controller |     - '    logger.go:42: 09:51:11 | watcher-tls/4-deploy-without-route | + ''['' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.816399 | controller |       '']'''
2026-03-14 09:56:26.816403 | controller |     - '    logger.go:42: 09:51:11 | watcher-tls/4-deploy-without-route | ++ oc get pods
2026-03-14 09:56:26.816408 | controller |       -n openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:56:26.816412 | controller |     - '    logger.go:42: 09:51:12 | watcher-tls/4-deploy-without-route | + ''['' ''''
2026-03-14 09:56:26.816416 | controller |       == '''' '']'''
2026-03-14 09:56:26.816420 | controller |     - '    logger.go:42: 09:51:12 | watcher-tls/4-deploy-without-route | + exit 0'
2026-03-14 09:56:26.816425 | controller |     - '    logger.go:42: 09:51:12 | watcher-tls/4-deploy-without-route | test step completed
2026-03-14 09:56:26.816429 | controller |       4-deploy-without-route'
2026-03-14 09:56:26.816433 | controller |     - '    logger.go:42: 09:51:12 | watcher-tls/5-disable-tls | starting test step 5-disable-tls'
2026-03-14 09:56:26.816437 | controller |     - '    logger.go:42: 09:51:12 | watcher-tls/5-disable-tls | running command: [sh -c
2026-03-14 09:56:26.816443 | controller |       oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op": "replace",
2026-03-14 09:56:26.816449 | controller |       "path": "/spec/apiServiceTemplate/override", "value":{}}]'''
2026-03-14 09:56:26.816453 | controller |     - '        ]'
2026-03-14 09:56:26.816458 | controller |     - '    logger.go:42: 09:51:12 | watcher-tls/5-disable-tls | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:56:26.816462 | controller |       patched'
2026-03-14 09:56:26.816466 | controller |     - '    logger.go:42: 09:51:12 | watcher-tls/5-disable-tls | running command: [sh -c
2026-03-14 09:56:26.816470 | controller |       set -euxo pipefail'
2026-03-14 09:56:26.816474 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:56:26.816479 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.816483 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.816487 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.816491 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.816496 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.816500 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.816504 | controller |       ]'
2026-03-14 09:56:26.816508 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.816513 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.816517 | controller |       == "" ]; then'
2026-03-14 09:56:26.816521 | controller |     - '            exit 0'
2026-03-14 09:56:26.816525 | controller |     - '        fi'
2026-03-14 09:56:26.816530 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.816534 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.816538 | controller |     - '        counter=0'
2026-03-14 09:56:26.816542 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.816547 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.816551 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.816555 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.816559 | controller |     - '            fi'
2026-03-14 09:56:26.816564 | controller |     - '        done'
2026-03-14 09:56:26.816568 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.816572 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.816576 | controller |     - '          exit 1'
2026-03-14 09:56:26.816580 | controller |     - '        else'
2026-03-14 09:56:26.816585 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.816589 | controller |     - '        fi'
2026-03-14 09:56:26.816604 | controller |     - '        # check that no watcher endpoint uses https'
2026-03-14 09:56:26.816611 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.816617 | controller |       infra-optim | [ $(grep -c https) == 0 ]'
2026-03-14 09:56:26.816623 | controller |     - '        ]'
2026-03-14 09:56:26.816628 | controller |     - '    logger.go:42: 09:51:12 | watcher-tls/5-disable-tls | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.816634 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:56:26.816639 | controller |     - '    logger.go:42: 09:51:12 | watcher-tls/5-disable-tls | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.816645 | controller |     - '    logger.go:42: 09:51:14 | watcher-tls/5-disable-tls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.816650 | controller |     - '    logger.go:42: 09:51:14 | watcher-tls/5-disable-tls | ++ grep watcher'
2026-03-14 09:56:26.816656 | controller |     - '    logger.go:42: 09:51:14 | watcher-tls/5-disable-tls | ++ awk ''{print $1}'''
2026-03-14 09:56:26.816661 | controller |     - '    logger.go:42: 09:51:14 | watcher-tls/5-disable-tls | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.816669 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:56:26.816675 | controller |     - '    logger.go:42: 09:51:16 | watcher-tls/5-disable-tls | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa'
2026-03-14 09:56:26.816680 | controller |     - '    logger.go:42: 09:51:16 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.816686 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.816704 | controller |     - '    logger.go:42: 09:51:16 | watcher-tls/5-disable-tls | + ''['' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.816711 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa '']'''
2026-03-14 09:56:26.816717 | controller |     - '    logger.go:42: 09:51:16 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.816722 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.816728 | controller |     - '    logger.go:42: 09:51:16 | watcher-tls/5-disable-tls | + ''['' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.816737 | controller |       '']'''
2026-03-14 09:56:26.816743 | controller |     - '    logger.go:42: 09:51:16 | watcher-tls/5-disable-tls | ++ oc get pods -n openstack-operators
2026-03-14 09:56:26.816749 | controller |       -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:56:26.816755 | controller |     - '    logger.go:42: 09:51:16 | watcher-tls/5-disable-tls | + ''['' '''' == '''' '']'''
2026-03-14 09:56:26.816761 | controller |     - '    logger.go:42: 09:51:16 | watcher-tls/5-disable-tls | + exit 0'
2026-03-14 09:56:26.816766 | controller |     - '    logger.go:42: 09:51:17 | watcher-tls/5-disable-tls | running command: [sh -c
2026-03-14 09:56:26.816772 | controller |       set -euxo pipefail'
2026-03-14 09:56:26.816778 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:56:26.816783 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.816789 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:56:26.816795 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.816800 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.816806 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.816812 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.816817 | controller |       ]'
2026-03-14 09:56:26.816823 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.816828 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.816834 | controller |       == "" ]; then'
2026-03-14 09:56:26.816839 | controller |     - '            exit 0'
2026-03-14 09:56:26.816845 | controller |     - '        fi'
2026-03-14 09:56:26.816851 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.816856 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.816862 | controller |     - '        counter=0'
2026-03-14 09:56:26.816868 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.816873 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.816879 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.816885 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.816890 | controller |     - '            fi'
2026-03-14 09:56:26.816896 | controller |     - '        done'
2026-03-14 09:56:26.816901 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.816907 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.816913 | controller |     - '          exit 1'
2026-03-14 09:56:26.816918 | controller |     - '        else'
2026-03-14 09:56:26.816924 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.816932 | controller |     - '        fi'
2026-03-14 09:56:26.816938 | controller |     - '        # check that no watcher endpoint uses https'
2026-03-14 09:56:26.816944 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:56:26.816949 | controller |       infra-optim | [ $(grep -c https) == 0 ]'
2026-03-14 09:56:26.816955 | controller |     - '        ]'
2026-03-14 09:56:26.816961 | controller |     - '    logger.go:42: 09:51:17 | watcher-tls/5-disable-tls | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.816967 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:56:26.816972 | controller |     - '    logger.go:42: 09:51:17 | watcher-tls/5-disable-tls | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.816977 | controller |     - '    logger.go:42: 09:51:19 | watcher-tls/5-disable-tls | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.816983 | controller |     - '    logger.go:42: 09:51:19 | watcher-tls/5-disable-tls | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.816988 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:56:26.816994 | controller |     - '    logger.go:42: 09:51:19 | watcher-tls/5-disable-tls | ++ awk ''{print $1}'''
2026-03-14 09:56:26.816999 | controller |     - '    logger.go:42: 09:51:19 | watcher-tls/5-disable-tls | ++ grep watcher'
2026-03-14 09:56:26.817005 | controller |     - '    logger.go:42: 09:51:21 | watcher-tls/5-disable-tls | + SERVICEID=bf6522b7dcd544e8aef389dfe89ae1fa'
2026-03-14 09:56:26.817010 | controller |     - '    logger.go:42: 09:51:21 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.817016 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.817021 | controller |     - '    logger.go:42: 09:51:21 | watcher-tls/5-disable-tls | + ''['' bf6522b7dcd544e8aef389dfe89ae1fa
2026-03-14 09:56:26.817027 | controller |       == bf6522b7dcd544e8aef389dfe89ae1fa '']'''
2026-03-14 09:56:26.817033 | controller |     - '    logger.go:42: 09:51:21 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.817038 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.817053 | controller |     - '    logger.go:42: 09:51:22 | watcher-tls/5-disable-tls | + ''['' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.817062 | controller |       '']'''
2026-03-14 09:56:26.817068 | controller |     - '    logger.go:42: 09:51:22 | watcher-tls/5-disable-tls | ++ oc get pods -n openstack-operators
2026-03-14 09:56:26.817073 | controller |       -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:56:26.817079 | controller |     - '    logger.go:42: 09:51:22 | watcher-tls/5-disable-tls | + ''['' '''' == '''' '']'''
2026-03-14 09:56:26.817084 | controller |     - '    logger.go:42: 09:51:22 | watcher-tls/5-disable-tls | + exit 0'
2026-03-14 09:56:26.817090 | controller |     - '    logger.go:42: 09:51:22 | watcher-tls/5-disable-tls | test step completed 5-disable-tls'
2026-03-14 09:56:26.817096 | controller |     - '    logger.go:42: 09:51:22 | watcher-tls/6-cleanup-watcher | starting test step
2026-03-14 09:56:26.817101 | controller |       6-cleanup-watcher'
2026-03-14 09:56:26.817106 | controller |     - '    logger.go:42: 09:51:22 | watcher-tls/6-cleanup-watcher | running command: [sh
2026-03-14 09:56:26.817112 | controller |       -c set -ex'
2026-03-14 09:56:26.817117 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.817122 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:56:26.817128 | controller |     - '        ]'
2026-03-14 09:56:26.817133 | controller |     - '    logger.go:42: 09:51:22 | watcher-tls/6-cleanup-watcher | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.817138 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:56:26.817144 | controller |     - '    logger.go:42: 09:51:22 | watcher-tls/6-cleanup-watcher | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.817150 | controller |     - '    logger.go:42: 09:51:24 | watcher-tls/6-cleanup-watcher | + ''['' 1 == 0 '']'''
2026-03-14 09:56:26.817155 | controller |     - '    logger.go:42: 09:51:26 | watcher-tls/6-cleanup-watcher | running command: [sh
2026-03-14 09:56:26.817163 | controller |       -c set -ex'
2026-03-14 09:56:26.817169 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.817174 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:56:26.817181 | controller |     - '        ]'
2026-03-14 09:56:26.817186 | controller |     - '    logger.go:42: 09:51:26 | watcher-tls/6-cleanup-watcher | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.817192 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:56:26.817197 | controller |     - '    logger.go:42: 09:51:26 | watcher-tls/6-cleanup-watcher | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.817203 | controller |     - '    logger.go:42: 09:51:28 | watcher-tls/6-cleanup-watcher | + ''['' 0 == 0 '']'''
2026-03-14 09:56:26.817208 | controller |     - '    logger.go:42: 09:51:28 | watcher-tls/6-cleanup-watcher | test step completed
2026-03-14 09:56:26.817214 | controller |       6-cleanup-watcher'
2026-03-14 09:56:26.817220 | controller |     - '    logger.go:42: 09:51:28 | watcher-tls/7-cleanup-certs | starting test step 7-cleanup-certs'
2026-03-14 09:56:26.817225 | controller |     - '    logger.go:42: 09:51:28 | watcher-tls/7-cleanup-certs | test step completed
2026-03-14 09:56:26.817231 | controller |       7-cleanup-certs'
2026-03-14 09:56:26.817236 | controller |     - '    logger.go:42: 09:51:28 | watcher-tls | skipping kubernetes event logging'
2026-03-14 09:56:26.817242 | controller |     - === CONT  kuttl/harness/watcher-rmquser
2026-03-14 09:56:26.817247 | controller |     - '    logger.go:42: 09:51:28 | watcher-rmquser | Skipping creation of user-supplied
2026-03-14 09:56:26.817252 | controller |       namespace: watcher-kuttl-default'
2026-03-14 09:56:26.817257 | controller |     - '    logger.go:42: 09:51:28 | watcher-rmquser/0-cleanup-watcher | starting test
2026-03-14 09:56:26.817263 | controller |       step 0-cleanup-watcher'
2026-03-14 09:56:26.817268 | controller |     - '    logger.go:42: 09:51:28 | watcher-rmquser/0-cleanup-watcher | test step completed
2026-03-14 09:56:26.817274 | controller |       0-cleanup-watcher'
2026-03-14 09:56:26.817280 | controller |     - '    logger.go:42: 09:51:28 | watcher-rmquser/1-deploy | starting test step 1-deploy'
2026-03-14 09:56:26.817285 | controller |     - '    logger.go:42: 09:51:28 | watcher-rmquser/1-deploy | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:56:26.817291 | controller |       created'
2026-03-14 09:56:26.817296 | controller |     - '    logger.go:42: 09:51:28 | watcher-rmquser/1-deploy | running command: [sh -c
2026-03-14 09:56:26.817302 | controller |       set -euxo pipefail'
2026-03-14 09:56:26.817307 | controller |     - '        '
2026-03-14 09:56:26.817312 | controller |     - '        # Wait for Watcher to be Ready'
2026-03-14 09:56:26.817318 | controller |     - '        kubectl wait --for=condition=Ready watcher/watcher-kuttl -n $NAMESPACE
2026-03-14 09:56:26.817326 | controller |       --timeout=300s'
2026-03-14 09:56:26.817331 | controller |     - '        '
2026-03-14 09:56:26.817337 | controller |     - '        # Verify WatcherNotificationTransportURLReady condition exists and is True'
2026-03-14 09:56:26.817342 | controller |     - '        kubectl get watcher watcher-kuttl -n $NAMESPACE -o jsonpath=''{.status.conditions[?(@.type=="WatcherNotificationTransportURLReady")].status}''
2026-03-14 09:56:26.817348 | controller |       | grep -q "True"'
2026-03-14 09:56:26.817353 | controller |     - '        echo "WatcherNotificationTransportURLReady condition is True"'
2026-03-14 09:56:26.817359 | controller |     - '        '
2026-03-14 09:56:26.817367 | controller |     - '        # Count TransportURL CRs - should be exactly 2 (one for messaging, one
2026-03-14 09:56:26.817372 | controller |       for notifications)'
2026-03-14 09:56:26.817378 | controller |     - '        transport_count=$(kubectl get transporturl -n $NAMESPACE -o name | grep
2026-03-14 09:56:26.817384 | controller |       "watcher-kuttl-watcher-transport" | wc -l)'
2026-03-14 09:56:26.817390 | controller |     - '        notification_transport_count=$(kubectl get transporturl -n $NAMESPACE -o
2026-03-14 09:56:26.817395 | controller |       name | grep "watcher-kuttl-watcher-notification" | wc -l)'
2026-03-14 09:56:26.817400 | controller |     - '        '
2026-03-14 09:56:26.817406 | controller |     - '        if [ "$transport_count" -ne "1" ]; then'
2026-03-14 09:56:26.817411 | controller |     - '          echo "Expected 1 watcher-transport TransportURL, found $transport_count"'
2026-03-14 09:56:26.817428 | controller |     - '          exit 1'
2026-03-14 09:56:26.817436 | controller |     - '        fi'
2026-03-14 09:56:26.817441 | controller |     - '        '
2026-03-14 09:56:26.817448 | controller |     - '        if [ "$notification_transport_count" -ne "1" ]; then'
2026-03-14 09:56:26.817454 | controller |     - '          echo "Expected 1 notification-transport TransportURL, found $notification_transport_count"'
2026-03-14 09:56:26.817459 | controller |     - '          exit 1'
2026-03-14 09:56:26.817465 | controller |     - '        fi'
2026-03-14 09:56:26.817470 | controller |     - '        '
2026-03-14 09:56:26.817476 | controller |     - '        echo "Correctly found 2 TransportURLs (separate clusters: transport and
2026-03-14 09:56:26.817482 | controller |       notification)"'
2026-03-14 09:56:26.817487 | controller |     - '        '
2026-03-14 09:56:26.817493 | controller |     - '        # Verify watcher-transport has correct user and vhost'
2026-03-14 09:56:26.817498 | controller |     - '        transport_user=$(kubectl get transporturl watcher-kuttl-watcher-transport
2026-03-14 09:56:26.817503 | controller |       -n $NAMESPACE -o jsonpath=''{.spec.username}'')'
2026-03-14 09:56:26.817509 | controller |     - '        transport_vhost=$(kubectl get transporturl watcher-kuttl-watcher-transport
2026-03-14 09:56:26.817514 | controller |       -n $NAMESPACE -o jsonpath=''{.spec.vhost}'')'
2026-03-14 09:56:26.817520 | controller |     - '        if [ "$transport_user" != "watcher-rpc" ]; then'
2026-03-14 09:56:26.817525 | controller |     - '          echo "Expected watcher-transport username ''watcher-rpc'', found ''$transport_user''"'
2026-03-14 09:56:26.817530 | controller |     - '          exit 1'
2026-03-14 09:56:26.817536 | controller |     - '        fi'
2026-03-14 09:56:26.817542 | controller |     - '        if [ "$transport_vhost" != "watcher-rpc" ]; then'
2026-03-14 09:56:26.817547 | controller |     - '          echo "Expected watcher-transport vhost ''watcher-rpc'', found ''$transport_vhost''"'
2026-03-14 09:56:26.817553 | controller |     - '          exit 1'
2026-03-14 09:56:26.817558 | controller |     - '        fi'
2026-03-14 09:56:26.817564 | controller |     - '        echo "Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)"'
2026-03-14 09:56:26.817570 | controller |     - '        '
2026-03-14 09:56:26.817575 | controller |     - '        # Verify notification-transport has correct user and vhost'
2026-03-14 09:56:26.817580 | controller |     - '        notif_user=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications
2026-03-14 09:56:26.817586 | controller |       -n $NAMESPACE -o jsonpath=''{.spec.username}'')'
2026-03-14 09:56:26.817591 | controller |     - '        notif_vhost=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications
2026-03-14 09:56:26.817617 | controller |       -n $NAMESPACE -o jsonpath=''{.spec.vhost}'')'
2026-03-14 09:56:26.817626 | controller |     - '        if [ "$notif_user" != "watcher-notifications" ]; then'
2026-03-14 09:56:26.817631 | controller |     - '          echo "Expected notification-transport username ''watcher-notifications'',
2026-03-14 09:56:26.817636 | controller |       found ''$notif_user''"'
2026-03-14 09:56:26.817641 | controller |     - '          exit 1'
2026-03-14 09:56:26.817647 | controller |     - '        fi'
2026-03-14 09:56:26.817652 | controller |     - '        if [ "$notif_vhost" != "watcher-notifications" ]; then'
2026-03-14 09:56:26.817657 | controller |     - '          echo "Expected notification-transport vhost ''watcher-notifications'',
2026-03-14 09:56:26.817662 | controller |       found ''$notif_vhost''"'
2026-03-14 09:56:26.817667 | controller |     - '          exit 1'
2026-03-14 09:56:26.817672 | controller |     - '        fi'
2026-03-14 09:56:26.817677 | controller |     - '        echo "Notification transport has correct user (watcher-notifications) and
2026-03-14 09:56:26.817684 | controller |       vhost (watcher-notifications)"'
2026-03-14 09:56:26.817691 | controller |     - '        '
2026-03-14 09:56:26.817696 | controller |     - '        # Verify that watcher.conf contains the notifications transport_url'
2026-03-14 09:56:26.817702 | controller |     - '        WATCHER_API_POD=$(kubectl get pods -n $NAMESPACE -l "service=watcher-api"
2026-03-14 09:56:26.817707 | controller |       -o custom-columns=:metadata.name --no-headers | grep -v ^$ | head -1)'
2026-03-14 09:56:26.817712 | controller |     - '        if [ -z "${WATCHER_API_POD}" ]; then'
2026-03-14 09:56:26.817721 | controller |     - '          echo "No watcher-api pod found"'
2026-03-14 09:56:26.817727 | controller |     - '          exit 1'
2026-03-14 09:56:26.817732 | controller |     - '        fi'
2026-03-14 09:56:26.817737 | controller |     - '        # Verify RPC transport_url in DEFAULT section'
2026-03-14 09:56:26.817742 | controller |     - '        rpc_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api
2026-03-14 09:56:26.817752 | controller |       -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -E ''^\[DEFAULT\]'' -A
2026-03-14 09:56:26.817757 | controller |       50 | grep ''transport_url'' | head -1 || true)'
2026-03-14 09:56:26.817762 | controller |     - '        if [ -z "$rpc_transport_url" ]; then'
2026-03-14 09:56:26.817768 | controller |     - '          echo "transport_url not found in DEFAULT section"'
2026-03-14 09:56:26.817773 | controller |     - '          exit 1'
2026-03-14 09:56:26.817779 | controller |     - '        fi'
2026-03-14 09:56:26.817784 | controller |     - '        echo "Found RPC transport_url: $rpc_transport_url"'
2026-03-14 09:56:26.817789 | controller |     - '        '
2026-03-14 09:56:26.817794 | controller |     - '        # Verify the RPC transport_url contains the correct vhost (watcher-rpc)'
2026-03-14 09:56:26.817799 | controller |     - '        if ! echo "$rpc_transport_url" | grep -q ''/watcher-rpc''; then'
2026-03-14 09:56:26.817805 | controller |     - '          echo "RPC transport_url does not contain expected vhost ''/watcher-rpc''"'
2026-03-14 09:56:26.817811 | controller |     - '          exit 1'
2026-03-14 09:56:26.817816 | controller |     - '        fi'
2026-03-14 09:56:26.817821 | controller |     - '        echo "Successfully verified vhost ''watcher-rpc'' in RPC transport_url"'
2026-03-14 09:56:26.817827 | controller |     - '        '
2026-03-14 09:56:26.817832 | controller |     - '        # Verify the RPC transport_url contains the correct username (watcher-rpc)'
2026-03-14 09:56:26.817837 | controller |     - '        if ! echo "$rpc_transport_url" | grep -q ''watcher-rpc:''; then'
2026-03-14 09:56:26.817843 | controller |     - '          echo "RPC transport_url does not contain expected username ''watcher-rpc:''"'
2026-03-14 09:56:26.817848 | controller |     - '          exit 1'
2026-03-14 09:56:26.817852 | controller |     - '        fi'
2026-03-14 09:56:26.817858 | controller |     - '        echo "Successfully verified username ''watcher-rpc'' in RPC transport_url"'
2026-03-14 09:56:26.817874 | controller |     - '        '
2026-03-14 09:56:26.817880 | controller |     - '        # Verify oslo_messaging_notifications section has transport_url configured'
2026-03-14 09:56:26.817886 | controller |     - '        notif_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c
2026-03-14 09:56:26.817891 | controller |       watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -A 5 ''\[oslo_messaging_notifications\]''
2026-03-14 09:56:26.817896 | controller |       | grep ''transport_url'' || true)'
2026-03-14 09:56:26.817901 | controller |     - '        if [ -z "$notif_transport_url" ]; then'
2026-03-14 09:56:26.817906 | controller |     - '          echo "transport_url not found in oslo_messaging_notifications section"'
2026-03-14 09:56:26.817913 | controller |     - '          exit 1'
2026-03-14 09:56:26.817918 | controller |     - '        fi'
2026-03-14 09:56:26.817923 | controller |     - '        echo "Found notifications transport_url: $notif_transport_url"'
2026-03-14 09:56:26.817928 | controller |     - '        '
2026-03-14 09:56:26.817934 | controller |     - '        # Verify the notifications transport_url contains the correct vhost (watcher-notifications)'
2026-03-14 09:56:26.817939 | controller |     - '        if ! echo "$notif_transport_url" | grep -q ''/watcher-notifications'';
2026-03-14 09:56:26.817944 | controller |       then'
2026-03-14 09:56:26.817949 | controller |     - '          echo "Notifications transport_url does not contain expected vhost ''/watcher-notifications''"'
2026-03-14 09:56:26.817957 | controller |     - '          exit 1'
2026-03-14 09:56:26.817963 | controller |     - '        fi'
2026-03-14 09:56:26.817968 | controller |     - '        echo "Successfully verified vhost ''watcher-notifications'' in notifications
2026-03-14 09:56:26.817973 | controller |       transport_url"'
2026-03-14 09:56:26.817981 | controller |     - '        '
2026-03-14 09:56:26.817987 | controller |     - '        # Verify the notifications transport_url contains the correct username
2026-03-14 09:56:26.817992 | controller |       (watcher-notifications)'
2026-03-14 09:56:26.817999 | controller |     - '        if ! echo "$notif_transport_url" | grep -q ''watcher-notifications:'';
2026-03-14 09:56:26.818005 | controller |       then'
2026-03-14 09:56:26.818010 | controller |     - '          echo "Notifications transport_url does not contain expected username
2026-03-14 09:56:26.818015 | controller |       ''watcher-notifications:''"'
2026-03-14 09:56:26.818023 | controller |     - '          exit 1'
2026-03-14 09:56:26.818028 | controller |     - '        fi'
2026-03-14 09:56:26.818034 | controller |     - '        echo "Successfully verified username ''watcher-notifications'' in notifications
2026-03-14 09:56:26.818039 | controller |       transport_url"'
2026-03-14 09:56:26.818044 | controller |     - '        '
2026-03-14 09:56:26.818049 | controller |     - '        exit 0'
2026-03-14 09:56:26.818055 | controller |     - '        ]'
2026-03-14 09:56:26.818060 | controller |     - '    logger.go:42: 09:51:28 | watcher-rmquser/1-deploy | + kubectl wait --for=condition=Ready
2026-03-14 09:56:26.818065 | controller |       watcher/watcher-kuttl -n watcher-kuttl-default --timeout=300s'
2026-03-14 09:56:26.818070 | controller |     - '    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:56:26.818075 | controller |       condition met'
2026-03-14 09:56:26.818081 | controller |     - '    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | + kubectl get watcher watcher-kuttl
2026-03-14 09:56:26.818086 | controller |       -n watcher-kuttl-default -o ''jsonpath={.status.conditions[?(@.type=="WatcherNotificationTransportURLReady")].status}'''
2026-03-14 09:56:26.818091 | controller |     - '    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | + grep -q True'
2026-03-14 09:56:26.818096 | controller |     - '    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | + echo ''WatcherNotificationTransportURLReady
2026-03-14 09:56:26.818101 | controller |       condition is True'''
2026-03-14 09:56:26.818108 | controller |     - '    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | WatcherNotificationTransportURLReady
2026-03-14 09:56:26.818114 | controller |       condition is True'
2026-03-14 09:56:26.818119 | controller |     - '    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | ++ wc -l'
2026-03-14 09:56:26.818124 | controller |     - '    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-transport'
2026-03-14 09:56:26.818129 | controller |     - '    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:56:26.818135 | controller |       -n watcher-kuttl-default -o name'
2026-03-14 09:56:26.818141 | controller |     - '    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | + transport_count=1'
2026-03-14 09:56:26.818147 | controller |     - '    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | ++ wc -l'
2026-03-14 09:56:26.818152 | controller |     - '    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-notification'
2026-03-14 09:56:26.818158 | controller |     - '    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:56:26.818163 | controller |       -n watcher-kuttl-default -o name'
2026-03-14 09:56:26.818168 | controller |     - '    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | + notification_transport_count=1'
2026-03-14 09:56:26.818174 | controller |     - '    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | + ''['' 1 -ne 1 '']'''
2026-03-14 09:56:26.818179 | controller |     - '    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | + ''['' 1 -ne 1 '']'''
2026-03-14 09:56:26.818184 | controller |     - '    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | + echo ''Correctly found
2026-03-14 09:56:26.818190 | controller |       2 TransportURLs (separate clusters: transport and notification)'''
2026-03-14 09:56:26.818195 | controller |     - '    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | Correctly found 2 TransportURLs
2026-03-14 09:56:26.818200 | controller |       (separate clusters: transport and notification)'
2026-03-14 09:56:26.818206 | controller |     - '    logger.go:42: 09:51:53 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:56:26.818211 | controller |       watcher-kuttl-watcher-transport -n watcher-kuttl-default -o ''jsonpath={.spec.username}'''
2026-03-14 09:56:26.818224 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + transport_user=watcher-rpc'
2026-03-14 09:56:26.818230 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:56:26.818238 | controller |       watcher-kuttl-watcher-transport -n watcher-kuttl-default -o ''jsonpath={.spec.vhost}'''
2026-03-14 09:56:26.818243 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + transport_vhost=watcher-rpc'
2026-03-14 09:56:26.818249 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + ''['' watcher-rpc ''!=''
2026-03-14 09:56:26.818254 | controller |       watcher-rpc '']'''
2026-03-14 09:56:26.818260 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + ''['' watcher-rpc ''!=''
2026-03-14 09:56:26.818265 | controller |       watcher-rpc '']'''
2026-03-14 09:56:26.818270 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + echo ''Watcher transport
2026-03-14 09:56:26.818276 | controller |       has correct user (watcher-rpc) and vhost (watcher-rpc)'''
2026-03-14 09:56:26.818281 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | Watcher transport has correct
2026-03-14 09:56:26.818286 | controller |       user (watcher-rpc) and vhost (watcher-rpc)'
2026-03-14 09:56:26.818291 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:56:26.818297 | controller |       watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default
2026-03-14 09:56:26.818302 | controller |       -o ''jsonpath={.spec.username}'''
2026-03-14 09:56:26.818308 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + notif_user=watcher-notifications'
2026-03-14 09:56:26.818313 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:56:26.818318 | controller |       watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default
2026-03-14 09:56:26.818323 | controller |       -o ''jsonpath={.spec.vhost}'''
2026-03-14 09:56:26.818329 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + notif_vhost=watcher-notifications'
2026-03-14 09:56:26.818334 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + ''['' watcher-notifications
2026-03-14 09:56:26.818339 | controller |       ''!='' watcher-notifications '']'''
2026-03-14 09:56:26.818345 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + ''['' watcher-notifications
2026-03-14 09:56:26.818350 | controller |       ''!='' watcher-notifications '']'''
2026-03-14 09:56:26.818355 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + echo ''Notification transport
2026-03-14 09:56:26.818361 | controller |       has correct user (watcher-notifications) and vhost (watcher-notifications)'''
2026-03-14 09:56:26.818366 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | Notification transport
2026-03-14 09:56:26.818371 | controller |       has correct user (watcher-notifications) and vhost (watcher-notifications)'
2026-03-14 09:56:26.818377 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | ++ grep -v ''^$'''
2026-03-14 09:56:26.818382 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | ++ kubectl get pods -n
2026-03-14 09:56:26.818387 | controller |       watcher-kuttl-default -l service=watcher-api -o custom-columns=:metadata.name --no-headers'
2026-03-14 09:56:26.818393 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | ++ head -1'
2026-03-14 09:56:26.818398 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + WATCHER_API_POD=watcher-kuttl-api-0'
2026-03-14 09:56:26.818403 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + ''['' -z watcher-kuttl-api-0
2026-03-14 09:56:26.818409 | controller |       '']'''
2026-03-14 09:56:26.818414 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | ++ head -1'
2026-03-14 09:56:26.818419 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | ++ grep transport_url'
2026-03-14 09:56:26.818425 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | ++ grep -E ''^\[DEFAULT\]''
2026-03-14 09:56:26.818430 | controller |       -A 50'
2026-03-14 09:56:26.818435 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default
2026-03-14 09:56:26.818441 | controller |       watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.818449 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + rpc_transport_url=''transport_url
2026-03-14 09:56:26.818454 | controller |       = rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'''
2026-03-14 09:56:26.818462 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + ''['' -z ''transport_url
2026-03-14 09:56:26.818467 | controller |       = rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1''
2026-03-14 09:56:26.818472 | controller |       '']'''
2026-03-14 09:56:26.818478 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + echo ''Found RPC transport_url:
2026-03-14 09:56:26.818484 | controller |       transport_url = rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'''
2026-03-14 09:56:26.818489 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | Found RPC transport_url:
2026-03-14 09:56:26.818494 | controller |       transport_url = rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'
2026-03-14 09:56:26.818500 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + grep -q /watcher-rpc'
2026-03-14 09:56:26.818513 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + echo ''transport_url
2026-03-14 09:56:26.818519 | controller |       = rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'''
2026-03-14 09:56:26.818524 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + echo ''Successfully verified
2026-03-14 09:56:26.818530 | controller |       vhost ''\''''watcher-rpc''\'''' in RPC transport_url'''
2026-03-14 09:56:26.818535 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | Successfully verified vhost
2026-03-14 09:56:26.818540 | controller |       ''watcher-rpc'' in RPC transport_url'
2026-03-14 09:56:26.818546 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + grep -q watcher-rpc:'
2026-03-14 09:56:26.818551 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + echo ''transport_url
2026-03-14 09:56:26.818556 | controller |       = rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'''
2026-03-14 09:56:26.818562 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | + echo ''Successfully verified
2026-03-14 09:56:26.818567 | controller |       username ''\''''watcher-rpc''\'''' in RPC transport_url'''
2026-03-14 09:56:26.818572 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | Successfully verified username
2026-03-14 09:56:26.818578 | controller |       ''watcher-rpc'' in RPC transport_url'
2026-03-14 09:56:26.818583 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | ++ grep -A 5 ''\[oslo_messaging_notifications\]'''
2026-03-14 09:56:26.818588 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | ++ grep transport_url'
2026-03-14 09:56:26.818605 | controller |     - '    logger.go:42: 09:51:54 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default
2026-03-14 09:56:26.818613 | controller |       watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.818618 | controller |     - '    logger.go:42: 09:51:55 | watcher-rmquser/1-deploy | + notif_transport_url=''transport_url
2026-03-14 09:56:26.818624 | controller |       = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'''
2026-03-14 09:56:26.818629 | controller |     - '    logger.go:42: 09:51:55 | watcher-rmquser/1-deploy | + ''['' -z ''transport_url
2026-03-14 09:56:26.818634 | controller |       = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1''
2026-03-14 09:56:26.818640 | controller |       '']'''
2026-03-14 09:56:26.818645 | controller |     - '    logger.go:42: 09:51:55 | watcher-rmquser/1-deploy | + echo ''Found notifications
2026-03-14 09:56:26.818654 | controller |       transport_url: transport_url = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'''
2026-03-14 09:56:26.818661 | controller |     - '    logger.go:42: 09:51:55 | watcher-rmquser/1-deploy | Found notifications transport_url:
2026-03-14 09:56:26.818667 | controller |       transport_url = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'
2026-03-14 09:56:26.818672 | controller |     - '    logger.go:42: 09:51:55 | watcher-rmquser/1-deploy | + grep -q /watcher-notifications'
2026-03-14 09:56:26.818678 | controller |     - '    logger.go:42: 09:51:55 | watcher-rmquser/1-deploy | + echo ''transport_url
2026-03-14 09:56:26.818683 | controller |       = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'''
2026-03-14 09:56:26.818688 | controller |     - '    logger.go:42: 09:51:55 | watcher-rmquser/1-deploy | + echo ''Successfully verified
2026-03-14 09:56:26.818694 | controller |       vhost ''\''''watcher-notifications''\'''' in notifications transport_url'''
2026-03-14 09:56:26.818699 | controller |     - '    logger.go:42: 09:51:55 | watcher-rmquser/1-deploy | Successfully verified vhost
2026-03-14 09:56:26.818704 | controller |       ''watcher-notifications'' in notifications transport_url'
2026-03-14 09:56:26.818709 | controller |     - '    logger.go:42: 09:51:55 | watcher-rmquser/1-deploy | + grep -q watcher-notifications:'
2026-03-14 09:56:26.818714 | controller |     - '    logger.go:42: 09:51:55 | watcher-rmquser/1-deploy | + echo ''transport_url
2026-03-14 09:56:26.818719 | controller |       = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'''
2026-03-14 09:56:26.818725 | controller |     - '    logger.go:42: 09:51:55 | watcher-rmquser/1-deploy | + echo ''Successfully verified
2026-03-14 09:56:26.818730 | controller |       username ''\''''watcher-notifications''\'''' in notifications transport_url'''
2026-03-14 09:56:26.818735 | controller |     - '    logger.go:42: 09:51:55 | watcher-rmquser/1-deploy | Successfully verified username
2026-03-14 09:56:26.818740 | controller |       ''watcher-notifications'' in notifications transport_url'
2026-03-14 09:56:26.818745 | controller |     - '    logger.go:42: 09:51:55 | watcher-rmquser/1-deploy | + exit 0'
2026-03-14 09:56:26.818750 | controller |     - '    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | running command: [sh -c
2026-03-14 09:56:26.818756 | controller |       set -euxo pipefail'
2026-03-14 09:56:26.818761 | controller |     - '        '
2026-03-14 09:56:26.818776 | controller |     - '        # Wait for Watcher to be Ready'
2026-03-14 09:56:26.818781 | controller |     - '        kubectl wait --for=condition=Ready watcher/watcher-kuttl -n $NAMESPACE
2026-03-14 09:56:26.818787 | controller |       --timeout=300s'
2026-03-14 09:56:26.818792 | controller |     - '        '
2026-03-14 09:56:26.818800 | controller |     - '        # Verify WatcherNotificationTransportURLReady condition exists and is True'
2026-03-14 09:56:26.818806 | controller |     - '        kubectl get watcher watcher-kuttl -n $NAMESPACE -o jsonpath=''{.status.conditions[?(@.type=="WatcherNotificationTransportURLReady")].status}''
2026-03-14 09:56:26.818811 | controller |       | grep -q "True"'
2026-03-14 09:56:26.818817 | controller |     - '        echo "WatcherNotificationTransportURLReady condition is True"'
2026-03-14 09:56:26.818822 | controller |     - '        '
2026-03-14 09:56:26.818827 | controller |     - '        # Count TransportURL CRs - should be exactly 2 (one for messaging, one
2026-03-14 09:56:26.818833 | controller |       for notifications)'
2026-03-14 09:56:26.818838 | controller |     - '        transport_count=$(kubectl get transporturl -n $NAMESPACE -o name | grep
2026-03-14 09:56:26.818844 | controller |       "watcher-kuttl-watcher-transport" | wc -l)'
2026-03-14 09:56:26.818849 | controller |     - '        notification_transport_count=$(kubectl get transporturl -n $NAMESPACE -o
2026-03-14 09:56:26.818854 | controller |       name | grep "watcher-kuttl-watcher-notification" | wc -l)'
2026-03-14 09:56:26.818859 | controller |     - '        '
2026-03-14 09:56:26.818867 | controller |     - '        if [ "$transport_count" -ne "1" ]; then'
2026-03-14 09:56:26.818872 | controller |     - '          echo "Expected 1 watcher-transport TransportURL, found $transport_count"'
2026-03-14 09:56:26.818881 | controller |     - '          exit 1'
2026-03-14 09:56:26.818887 | controller |     - '        fi'
2026-03-14 09:56:26.818892 | controller |     - '        '
2026-03-14 09:56:26.818897 | controller |     - '        if [ "$notification_transport_count" -ne "1" ]; then'
2026-03-14 09:56:26.818902 | controller |     - '          echo "Expected 1 notification-transport TransportURL, found $notification_transport_count"'
2026-03-14 09:56:26.818907 | controller |     - '          exit 1'
2026-03-14 09:56:26.818912 | controller |     - '        fi'
2026-03-14 09:56:26.818917 | controller |     - '        '
2026-03-14 09:56:26.818922 | controller |     - '        echo "Correctly found 2 TransportURLs (separate clusters: transport and
2026-03-14 09:56:26.818927 | controller |       notification)"'
2026-03-14 09:56:26.818932 | controller |     - '        '
2026-03-14 09:56:26.818937 | controller |     - '        # Verify watcher-transport has correct user and vhost'
2026-03-14 09:56:26.818943 | controller |     - '        transport_user=$(kubectl get transporturl watcher-kuttl-watcher-transport
2026-03-14 09:56:26.818948 | controller |       -n $NAMESPACE -o jsonpath=''{.spec.username}'')'
2026-03-14 09:56:26.818955 | controller |     - '        transport_vhost=$(kubectl get transporturl watcher-kuttl-watcher-transport
2026-03-14 09:56:26.818960 | controller |       -n $NAMESPACE -o jsonpath=''{.spec.vhost}'')'
2026-03-14 09:56:26.818966 | controller |     - '        if [ "$transport_user" != "watcher-rpc" ]; then'
2026-03-14 09:56:26.818971 | controller |     - '          echo "Expected watcher-transport username ''watcher-rpc'', found ''$transport_user''"'
2026-03-14 09:56:26.818977 | controller |     - '          exit 1'
2026-03-14 09:56:26.818982 | controller |     - '        fi'
2026-03-14 09:56:26.818987 | controller |     - '        if [ "$transport_vhost" != "watcher-rpc" ]; then'
2026-03-14 09:56:26.818993 | controller |     - '          echo "Expected watcher-transport vhost ''watcher-rpc'', found ''$transport_vhost''"'
2026-03-14 09:56:26.818998 | controller |     - '          exit 1'
2026-03-14 09:56:26.819003 | controller |     - '        fi'
2026-03-14 09:56:26.819009 | controller |     - '        echo "Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)"'
2026-03-14 09:56:26.819014 | controller |     - '        '
2026-03-14 09:56:26.819020 | controller |     - '        # Verify notification-transport has correct user and vhost'
2026-03-14 09:56:26.819025 | controller |     - '        notif_user=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications
2026-03-14 09:56:26.819030 | controller |       -n $NAMESPACE -o jsonpath=''{.spec.username}'')'
2026-03-14 09:56:26.819036 | controller |     - '        notif_vhost=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications
2026-03-14 09:56:26.819041 | controller |       -n $NAMESPACE -o jsonpath=''{.spec.vhost}'')'
2026-03-14 09:56:26.819061 | controller |     - '        if [ "$notif_user" != "watcher-notifications" ]; then'
2026-03-14 09:56:26.819067 | controller |     - '          echo "Expected notification-transport username ''watcher-notifications'',
2026-03-14 09:56:26.819072 | controller |       found ''$notif_user''"'
2026-03-14 09:56:26.819078 | controller |     - '          exit 1'
2026-03-14 09:56:26.819083 | controller |     - '        fi'
2026-03-14 09:56:26.819089 | controller |     - '        if [ "$notif_vhost" != "watcher-notifications" ]; then'
2026-03-14 09:56:26.819094 | controller |     - '          echo "Expected notification-transport vhost ''watcher-notifications'',
2026-03-14 09:56:26.819099 | controller |       found ''$notif_vhost''"'
2026-03-14 09:56:26.819105 | controller |     - '          exit 1'
2026-03-14 09:56:26.819110 | controller |     - '        fi'
2026-03-14 09:56:26.819116 | controller |     - '        echo "Notification transport has correct user (watcher-notifications) and
2026-03-14 09:56:26.819121 | controller |       vhost (watcher-notifications)"'
2026-03-14 09:56:26.819126 | controller |     - '        '
2026-03-14 09:56:26.819132 | controller |     - '        # Verify that watcher.conf contains the notifications transport_url'
2026-03-14 09:56:26.819137 | controller |     - '        WATCHER_API_POD=$(kubectl get pods -n $NAMESPACE -l "service=watcher-api"
2026-03-14 09:56:26.819142 | controller |       -o custom-columns=:metadata.name --no-headers | grep -v ^$ | head -1)'
2026-03-14 09:56:26.819150 | controller |     - '        if [ -z "${WATCHER_API_POD}" ]; then'
2026-03-14 09:56:26.819156 | controller |     - '          echo "No watcher-api pod found"'
2026-03-14 09:56:26.819161 | controller |     - '          exit 1'
2026-03-14 09:56:26.819167 | controller |     - '        fi'
2026-03-14 09:56:26.819172 | controller |     - '        # Verify RPC transport_url in DEFAULT section'
2026-03-14 09:56:26.819177 | controller |     - '        rpc_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api
2026-03-14 09:56:26.819183 | controller |       -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -E ''^\[DEFAULT\]'' -A
2026-03-14 09:56:26.819196 | controller |       50 | grep ''transport_url'' | head -1 || true)'
2026-03-14 09:56:26.819203 | controller |     - '        if [ -z "$rpc_transport_url" ]; then'
2026-03-14 09:56:26.819209 | controller |     - '          echo "transport_url not found in DEFAULT section"'
2026-03-14 09:56:26.819214 | controller |     - '          exit 1'
2026-03-14 09:56:26.819221 | controller |     - '        fi'
2026-03-14 09:56:26.819226 | controller |     - '        echo "Found RPC transport_url: $rpc_transport_url"'
2026-03-14 09:56:26.819231 | controller |     - '        '
2026-03-14 09:56:26.819236 | controller |     - '        # Verify the RPC transport_url contains the correct vhost (watcher-rpc)'
2026-03-14 09:56:26.819242 | controller |     - '        if ! echo "$rpc_transport_url" | grep -q ''/watcher-rpc''; then'
2026-03-14 09:56:26.819247 | controller |     - '          echo "RPC transport_url does not contain expected vhost ''/watcher-rpc''"'
2026-03-14 09:56:26.819252 | controller |     - '          exit 1'
2026-03-14 09:56:26.819257 | controller |     - '        fi'
2026-03-14 09:56:26.819262 | controller |     - '        echo "Successfully verified vhost ''watcher-rpc'' in RPC transport_url"'
2026-03-14 09:56:26.819267 | controller |     - '        '
2026-03-14 09:56:26.819273 | controller |     - '        # Verify the RPC transport_url contains the correct username (watcher-rpc)'
2026-03-14 09:56:26.819278 | controller |     - '        if ! echo "$rpc_transport_url" | grep -q ''watcher-rpc:''; then'
2026-03-14 09:56:26.819283 | controller |     - '          echo "RPC transport_url does not contain expected username ''watcher-rpc:''"'
2026-03-14 09:56:26.819289 | controller |     - '          exit 1'
2026-03-14 09:56:26.819294 | controller |     - '        fi'
2026-03-14 09:56:26.819299 | controller |     - '        echo "Successfully verified username ''watcher-rpc'' in RPC transport_url"'
2026-03-14 09:56:26.819304 | controller |     - '        '
2026-03-14 09:56:26.819310 | controller |     - '        # Verify oslo_messaging_notifications section has transport_url configured'
2026-03-14 09:56:26.819315 | controller |     - '        notif_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c
2026-03-14 09:56:26.819320 | controller |       watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -A 5 ''\[oslo_messaging_notifications\]''
2026-03-14 09:56:26.819326 | controller |       | grep ''transport_url'' || true)'
2026-03-14 09:56:26.819331 | controller |     - '        if [ -z "$notif_transport_url" ]; then'
2026-03-14 09:56:26.819336 | controller |     - '          echo "transport_url not found in oslo_messaging_notifications section"'
2026-03-14 09:56:26.819342 | controller |     - '          exit 1'
2026-03-14 09:56:26.819347 | controller |     - '        fi'
2026-03-14 09:56:26.819352 | controller |     - '        echo "Found notifications transport_url: $notif_transport_url"'
2026-03-14 09:56:26.819358 | controller |     - '        '
2026-03-14 09:56:26.819363 | controller |     - '        # Verify the notifications transport_url contains the correct vhost (watcher-notifications)'
2026-03-14 09:56:26.819368 | controller |     - '        if ! echo "$notif_transport_url" | grep -q ''/watcher-notifications'';
2026-03-14 09:56:26.819374 | controller |       then'
2026-03-14 09:56:26.819379 | controller |     - '          echo "Notifications transport_url does not contain expected vhost ''/watcher-notifications''"'
2026-03-14 09:56:26.819388 | controller |     - '          exit 1'
2026-03-14 09:56:26.819394 | controller |     - '        fi'
2026-03-14 09:56:26.819399 | controller |     - '        echo "Successfully verified vhost ''watcher-notifications'' in notifications
2026-03-14 09:56:26.819404 | controller |       transport_url"'
2026-03-14 09:56:26.819409 | controller |     - '        '
2026-03-14 09:56:26.819414 | controller |     - '        # Verify the notifications transport_url contains the correct username
2026-03-14 09:56:26.819423 | controller |       (watcher-notifications)'
2026-03-14 09:56:26.819428 | controller |     - '        if ! echo "$notif_transport_url" | grep -q ''watcher-notifications:'';
2026-03-14 09:56:26.819433 | controller |       then'
2026-03-14 09:56:26.819439 | controller |     - '          echo "Notifications transport_url does not contain expected username
2026-03-14 09:56:26.819444 | controller |       ''watcher-notifications:''"'
2026-03-14 09:56:26.819449 | controller |     - '          exit 1'
2026-03-14 09:56:26.819454 | controller |     - '        fi'
2026-03-14 09:56:26.819459 | controller |     - '        echo "Successfully verified username ''watcher-notifications'' in notifications
2026-03-14 09:56:26.819465 | controller |       transport_url"'
2026-03-14 09:56:26.819470 | controller |     - '        '
2026-03-14 09:56:26.819475 | controller |     - '        exit 0'
2026-03-14 09:56:26.819480 | controller |     - '        ]'
2026-03-14 09:56:26.819485 | controller |     - '    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | + kubectl wait --for=condition=Ready
2026-03-14 09:56:26.819491 | controller |       watcher/watcher-kuttl -n watcher-kuttl-default --timeout=300s'
2026-03-14 09:56:26.819496 | controller |     - '    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:56:26.819501 | controller |       condition met'
2026-03-14 09:56:26.819507 | controller |     - '    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | + kubectl get watcher watcher-kuttl
2026-03-14 09:56:26.819512 | controller |       -n watcher-kuttl-default -o ''jsonpath={.status.conditions[?(@.type=="WatcherNotificationTransportURLReady")].status}'''
2026-03-14 09:56:26.819517 | controller |     - '    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | + grep -q True'
2026-03-14 09:56:26.819523 | controller |     - '    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | + echo ''WatcherNotificationTransportURLReady
2026-03-14 09:56:26.819528 | controller |       condition is True'''
2026-03-14 09:56:26.819536 | controller |     - '    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | WatcherNotificationTransportURLReady
2026-03-14 09:56:26.819542 | controller |       condition is True'
2026-03-14 09:56:26.819547 | controller |     - '    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:56:26.819553 | controller |       -n watcher-kuttl-default -o name'
2026-03-14 09:56:26.819558 | controller |     - '    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-transport'
2026-03-14 09:56:26.819563 | controller |     - '    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | ++ wc -l'
2026-03-14 09:56:26.819568 | controller |     - '    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | + transport_count=1'
2026-03-14 09:56:26.819585 | controller |     - '    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-notification'
2026-03-14 09:56:26.819590 | controller |     - '    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | ++ wc -l'
2026-03-14 09:56:26.819621 | controller |     - '    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:56:26.819629 | controller |       -n watcher-kuttl-default -o name'
2026-03-14 09:56:26.819634 | controller |     - '    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | + notification_transport_count=1'
2026-03-14 09:56:26.819639 | controller |     - '    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | + ''['' 1 -ne 1 '']'''
2026-03-14 09:56:26.819644 | controller |     - '    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | + ''['' 1 -ne 1 '']'''
2026-03-14 09:56:26.819649 | controller |     - '    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | + echo ''Correctly found
2026-03-14 09:56:26.819654 | controller |       2 TransportURLs (separate clusters: transport and notification)'''
2026-03-14 09:56:26.819659 | controller |     - '    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | Correctly found 2 TransportURLs
2026-03-14 09:56:26.819664 | controller |       (separate clusters: transport and notification)'
2026-03-14 09:56:26.819672 | controller |     - '    logger.go:42: 09:51:56 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:56:26.819677 | controller |       watcher-kuttl-watcher-transport -n watcher-kuttl-default -o ''jsonpath={.spec.username}'''
2026-03-14 09:56:26.819682 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + transport_user=watcher-rpc'
2026-03-14 09:56:26.819690 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:56:26.819695 | controller |       watcher-kuttl-watcher-transport -n watcher-kuttl-default -o ''jsonpath={.spec.vhost}'''
2026-03-14 09:56:26.819700 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + transport_vhost=watcher-rpc'
2026-03-14 09:56:26.819705 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + ''['' watcher-rpc ''!=''
2026-03-14 09:56:26.819710 | controller |       watcher-rpc '']'''
2026-03-14 09:56:26.819715 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + ''['' watcher-rpc ''!=''
2026-03-14 09:56:26.819720 | controller |       watcher-rpc '']'''
2026-03-14 09:56:26.819725 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo ''Watcher transport
2026-03-14 09:56:26.819731 | controller |       has correct user (watcher-rpc) and vhost (watcher-rpc)'''
2026-03-14 09:56:26.819736 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | Watcher transport has correct
2026-03-14 09:56:26.819741 | controller |       user (watcher-rpc) and vhost (watcher-rpc)'
2026-03-14 09:56:26.819746 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:56:26.819752 | controller |       watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default
2026-03-14 09:56:26.819757 | controller |       -o ''jsonpath={.spec.username}'''
2026-03-14 09:56:26.819762 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + notif_user=watcher-notifications'
2026-03-14 09:56:26.819767 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:56:26.819773 | controller |       watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default
2026-03-14 09:56:26.819778 | controller |       -o ''jsonpath={.spec.vhost}'''
2026-03-14 09:56:26.819783 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + notif_vhost=watcher-notifications'
2026-03-14 09:56:26.819788 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + ''['' watcher-notifications
2026-03-14 09:56:26.819794 | controller |       ''!='' watcher-notifications '']'''
2026-03-14 09:56:26.819799 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + ''['' watcher-notifications
2026-03-14 09:56:26.819804 | controller |       ''!='' watcher-notifications '']'''
2026-03-14 09:56:26.819809 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo ''Notification transport
2026-03-14 09:56:26.819815 | controller |       has correct user (watcher-notifications) and vhost (watcher-notifications)'''
2026-03-14 09:56:26.819821 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | Notification transport
2026-03-14 09:56:26.819826 | controller |       has correct user (watcher-notifications) and vhost (watcher-notifications)'
2026-03-14 09:56:26.819831 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | ++ grep -v ''^$'''
2026-03-14 09:56:26.819836 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | ++ head -1'
2026-03-14 09:56:26.819841 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | ++ kubectl get pods -n
2026-03-14 09:56:26.819846 | controller |       watcher-kuttl-default -l service=watcher-api -o custom-columns=:metadata.name --no-headers'
2026-03-14 09:56:26.819851 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + WATCHER_API_POD=watcher-kuttl-api-0'
2026-03-14 09:56:26.819856 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + ''['' -z watcher-kuttl-api-0
2026-03-14 09:56:26.819862 | controller |       '']'''
2026-03-14 09:56:26.819867 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | ++ grep -E ''^\[DEFAULT\]''
2026-03-14 09:56:26.819872 | controller |       -A 50'
2026-03-14 09:56:26.819877 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | ++ grep transport_url'
2026-03-14 09:56:26.819882 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default
2026-03-14 09:56:26.819890 | controller |       watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.819900 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | ++ head -1'
2026-03-14 09:56:26.819905 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + rpc_transport_url=''transport_url
2026-03-14 09:56:26.819910 | controller |       = rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'''
2026-03-14 09:56:26.819915 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + ''['' -z ''transport_url
2026-03-14 09:56:26.819921 | controller |       = rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1''
2026-03-14 09:56:26.819926 | controller |       '']'''
2026-03-14 09:56:26.819931 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo ''Found RPC transport_url:
2026-03-14 09:56:26.819936 | controller |       transport_url = rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'''
2026-03-14 09:56:26.819941 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | Found RPC transport_url:
2026-03-14 09:56:26.819947 | controller |       transport_url = rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'
2026-03-14 09:56:26.819952 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo ''transport_url
2026-03-14 09:56:26.819957 | controller |       = rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'''
2026-03-14 09:56:26.819963 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + grep -q /watcher-rpc'
2026-03-14 09:56:26.819968 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo ''Successfully verified
2026-03-14 09:56:26.819973 | controller |       vhost ''\''''watcher-rpc''\'''' in RPC transport_url'''
2026-03-14 09:56:26.819978 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | Successfully verified vhost
2026-03-14 09:56:26.819988 | controller |       ''watcher-rpc'' in RPC transport_url'
2026-03-14 09:56:26.819996 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + grep -q watcher-rpc:'
2026-03-14 09:56:26.820001 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo ''transport_url
2026-03-14 09:56:26.820006 | controller |       = rabbit://watcher-rpc:VR2l5ofkub_61Xf-kR405af7FhYGLQQH@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'''
2026-03-14 09:56:26.820011 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo ''Successfully verified
2026-03-14 09:56:26.820016 | controller |       username ''\''''watcher-rpc''\'''' in RPC transport_url'''
2026-03-14 09:56:26.820021 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | Successfully verified username
2026-03-14 09:56:26.820026 | controller |       ''watcher-rpc'' in RPC transport_url'
2026-03-14 09:56:26.820031 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | ++ grep -A 5 ''\[oslo_messaging_notifications\]'''
2026-03-14 09:56:26.820036 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default
2026-03-14 09:56:26.820041 | controller |       watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.820047 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | ++ grep transport_url'
2026-03-14 09:56:26.820052 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + notif_transport_url=''transport_url
2026-03-14 09:56:26.820057 | controller |       = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'''
2026-03-14 09:56:26.820065 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + ''['' -z ''transport_url
2026-03-14 09:56:26.820070 | controller |       = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1''
2026-03-14 09:56:26.820077 | controller |       '']'''
2026-03-14 09:56:26.820082 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo ''Found notifications
2026-03-14 09:56:26.820087 | controller |       transport_url: transport_url = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'''
2026-03-14 09:56:26.820095 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | Found notifications transport_url:
2026-03-14 09:56:26.820100 | controller |       transport_url = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'
2026-03-14 09:56:26.820105 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + grep -q /watcher-notifications'
2026-03-14 09:56:26.820110 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo ''transport_url
2026-03-14 09:56:26.820115 | controller |       = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'''
2026-03-14 09:56:26.820120 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo ''Successfully verified
2026-03-14 09:56:26.820138 | controller |       vhost ''\''''watcher-notifications''\'''' in notifications transport_url'''
2026-03-14 09:56:26.820143 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | Successfully verified vhost
2026-03-14 09:56:26.820149 | controller |       ''watcher-notifications'' in notifications transport_url'
2026-03-14 09:56:26.820154 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + grep -q watcher-notifications:'
2026-03-14 09:56:26.820159 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo ''transport_url
2026-03-14 09:56:26.820165 | controller |       = rabbit://watcher-notifications:H1qrtw4jk3dupF_CafN8rQyZCSBgDORb@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'''
2026-03-14 09:56:26.820170 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + echo ''Successfully verified
2026-03-14 09:56:26.820175 | controller |       username ''\''''watcher-notifications''\'''' in notifications transport_url'''
2026-03-14 09:56:26.820181 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | Successfully verified username
2026-03-14 09:56:26.820186 | controller |       ''watcher-notifications'' in notifications transport_url'
2026-03-14 09:56:26.820191 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | + exit 0'
2026-03-14 09:56:26.820197 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/1-deploy | test step completed 1-deploy'
2026-03-14 09:56:26.820202 | controller |     - '    logger.go:42: 09:51:57 | watcher-rmquser/2-cleanup-watcher | starting test
2026-03-14 09:56:26.820207 | controller |       step 2-cleanup-watcher'
2026-03-14 09:56:26.820212 | controller |     - '    logger.go:42: 09:52:03 | watcher-rmquser/2-cleanup-watcher | test step completed
2026-03-14 09:56:26.820218 | controller |       2-cleanup-watcher'
2026-03-14 09:56:26.820223 | controller |     - '    logger.go:42: 09:52:03 | watcher-rmquser | skipping kubernetes event logging'
2026-03-14 09:56:26.820228 | controller |     - === CONT  kuttl/harness/watcher
2026-03-14 09:56:26.820234 | controller |     - '    logger.go:42: 09:52:03 | watcher | Skipping creation of user-supplied namespace:
2026-03-14 09:56:26.820239 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.820244 | controller |     - '    logger.go:42: 09:52:03 | watcher/0-cleanup-watcher | starting test step 0-cleanup-watcher'
2026-03-14 09:56:26.820249 | controller |     - '    logger.go:42: 09:52:03 | watcher/0-cleanup-watcher | test step completed 0-cleanup-watcher'
2026-03-14 09:56:26.820255 | controller |     - '    logger.go:42: 09:52:03 | watcher/1-deploy-with-defaults | starting test step
2026-03-14 09:56:26.820260 | controller |       1-deploy-with-defaults'
2026-03-14 09:56:26.820265 | controller |     - '    logger.go:42: 09:52:03 | watcher/1-deploy-with-defaults | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:56:26.820270 | controller |       created'
2026-03-14 09:56:26.820275 | controller |     - '    logger.go:42: 09:52:04 | watcher/1-deploy-with-defaults | running command:
2026-03-14 09:56:26.820282 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.820288 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.820293 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.820298 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:56:26.820303 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.820308 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.820313 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.820318 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.820323 | controller |       ]'
2026-03-14 09:56:26.820328 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.820336 | controller |       -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:56:26.820341 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.820346 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')"
2026-03-14 09:56:26.820351 | controller |       == 2 ]'
2026-03-14 09:56:26.820356 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.820361 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')"
2026-03-14 09:56:26.820366 | controller |       -ge 1 ]'
2026-03-14 09:56:26.820371 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.820376 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.820381 | controller |       == "" ]; then'
2026-03-14 09:56:26.820386 | controller |     - '            exit 0'
2026-03-14 09:56:26.820391 | controller |     - '        fi'
2026-03-14 09:56:26.820396 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.820401 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.820406 | controller |     - '        counter=0'
2026-03-14 09:56:26.820412 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.820417 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.820422 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.820427 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.820432 | controller |     - '            fi'
2026-03-14 09:56:26.820438 | controller |     - '        done'
2026-03-14 09:56:26.820443 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.820448 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.820462 | controller |     - '          exit 1'
2026-03-14 09:56:26.820468 | controller |     - '        else'
2026-03-14 09:56:26.820473 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.820478 | controller |     - '        fi'
2026-03-14 09:56:26.820484 | controller |     - '        ]'
2026-03-14 09:56:26.820489 | controller |     - '    logger.go:42: 09:52:04 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.820494 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:56:26.820500 | controller |     - '    logger.go:42: 09:52:04 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.820516 | controller |     - '    logger.go:42: 09:52:06 | watcher/1-deploy-with-defaults | + ''['' 0 == 1 '']'''
2026-03-14 09:56:26.820522 | controller |     - '    logger.go:42: 09:52:07 | watcher/1-deploy-with-defaults | running command:
2026-03-14 09:56:26.820527 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.820533 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.820541 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.820546 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:56:26.820551 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.820557 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.820562 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.820567 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.820573 | controller |       ]'
2026-03-14 09:56:26.820578 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.820583 | controller |       -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:56:26.820588 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.820605 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')"
2026-03-14 09:56:26.820613 | controller |       == 2 ]'
2026-03-14 09:56:26.820619 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.820624 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')"
2026-03-14 09:56:26.820629 | controller |       -ge 1 ]'
2026-03-14 09:56:26.820634 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.820640 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.820645 | controller |       == "" ]; then'
2026-03-14 09:56:26.820650 | controller |     - '            exit 0'
2026-03-14 09:56:26.820656 | controller |     - '        fi'
2026-03-14 09:56:26.820661 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.820666 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.820671 | controller |     - '        counter=0'
2026-03-14 09:56:26.820679 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.820685 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.820690 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.820695 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.820700 | controller |     - '            fi'
2026-03-14 09:56:26.820708 | controller |     - '        done'
2026-03-14 09:56:26.820713 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.820718 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.820723 | controller |     - '          exit 1'
2026-03-14 09:56:26.820728 | controller |     - '        else'
2026-03-14 09:56:26.820733 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.820739 | controller |     - '        fi'
2026-03-14 09:56:26.820744 | controller |     - '        ]'
2026-03-14 09:56:26.820750 | controller |     - '    logger.go:42: 09:52:07 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.820755 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:56:26.820760 | controller |     - '    logger.go:42: 09:52:07 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.820765 | controller |     - '    logger.go:42: 09:52:09 | watcher/1-deploy-with-defaults | + ''['' 0 == 1 '']'''
2026-03-14 09:56:26.820770 | controller |     - '    logger.go:42: 09:52:10 | watcher/1-deploy-with-defaults | running command:
2026-03-14 09:56:26.820775 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.820780 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.820785 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.820790 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:56:26.820806 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.820812 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.820817 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.820822 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.820827 | controller |       ]'
2026-03-14 09:56:26.820833 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.820838 | controller |       -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:56:26.820843 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.820857 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')"
2026-03-14 09:56:26.820863 | controller |       == 2 ]'
2026-03-14 09:56:26.820868 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.820873 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')"
2026-03-14 09:56:26.820878 | controller |       -ge 1 ]'
2026-03-14 09:56:26.820884 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.820889 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.820894 | controller |       == "" ]; then'
2026-03-14 09:56:26.820900 | controller |     - '            exit 0'
2026-03-14 09:56:26.820905 | controller |     - '        fi'
2026-03-14 09:56:26.820910 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.820915 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.820921 | controller |     - '        counter=0'
2026-03-14 09:56:26.820926 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.820931 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.820936 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.820941 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.820946 | controller |     - '            fi'
2026-03-14 09:56:26.820951 | controller |     - '        done'
2026-03-14 09:56:26.820956 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.820971 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.820976 | controller |     - '          exit 1'
2026-03-14 09:56:26.820981 | controller |     - '        else'
2026-03-14 09:56:26.820986 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.820991 | controller |     - '        fi'
2026-03-14 09:56:26.820996 | controller |     - '        ]'
2026-03-14 09:56:26.821001 | controller |     - '    logger.go:42: 09:52:10 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.821006 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:56:26.821015 | controller |     - '    logger.go:42: 09:52:10 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.821021 | controller |     - '    logger.go:42: 09:52:13 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.821026 | controller |     - '    logger.go:42: 09:52:13 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.821032 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:56:26.821037 | controller |     - '    logger.go:42: 09:52:13 | watcher/1-deploy-with-defaults | ++ grep watcher'
2026-03-14 09:56:26.821042 | controller |     - '    logger.go:42: 09:52:13 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}'''
2026-03-14 09:56:26.821047 | controller |     - '    logger.go:42: 09:52:15 | watcher/1-deploy-with-defaults | + SERVICEID=e32ab6f2cb3c4ec69d47af38a3e246ce'
2026-03-14 09:56:26.821052 | controller |     - '    logger.go:42: 09:52:15 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.821059 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.821064 | controller |     - '    logger.go:42: 09:52:15 | watcher/1-deploy-with-defaults | + ''['' e32ab6f2cb3c4ec69d47af38a3e246ce
2026-03-14 09:56:26.821069 | controller |       == e32ab6f2cb3c4ec69d47af38a3e246ce '']'''
2026-03-14 09:56:26.821074 | controller |     - '    logger.go:42: 09:52:15 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.821080 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.821085 | controller |     - '    logger.go:42: 09:52:15 | watcher/1-deploy-with-defaults | + ''['' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.821092 | controller |       '']'''
2026-03-14 09:56:26.821097 | controller |     - '    logger.go:42: 09:52:15 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:56:26.821102 | controller |     - '    logger.go:42: 09:52:15 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.821107 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}'''
2026-03-14 09:56:26.821112 | controller |     - '    logger.go:42: 09:52:15 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1'
2026-03-14 09:56:26.821118 | controller |     - '    logger.go:42: 09:52:16 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.821123 | controller |     - '    logger.go:42: 09:52:16 | watcher/1-deploy-with-defaults | ++ grep -c ''cafile
2026-03-14 09:56:26.821128 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'''
2026-03-14 09:56:26.821133 | controller |     - '    logger.go:42: 09:52:16 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:56:26.821139 | controller |     - '    logger.go:42: 09:52:16 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.821144 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:56:26.821149 | controller |     - '    logger.go:42: 09:52:16 | watcher/1-deploy-with-defaults | + ''['' 2 == 2 '']'''
2026-03-14 09:56:26.821154 | controller |     - '    logger.go:42: 09:52:16 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:56:26.821160 | controller |     - '    logger.go:42: 09:52:16 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.821165 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:56:26.821179 | controller |     - '    logger.go:42: 09:52:16 | watcher/1-deploy-with-defaults | ++ grep -c ''region_name
2026-03-14 09:56:26.821185 | controller |       = regionOne'''
2026-03-14 09:56:26.821191 | controller |     - '    logger.go:42: 09:52:16 | watcher/1-deploy-with-defaults | + ''['' 9 -ge 1 '']'''
2026-03-14 09:56:26.821196 | controller |     - '    logger.go:42: 09:52:16 | watcher/1-deploy-with-defaults | ++ oc get pods -n
2026-03-14 09:56:26.821201 | controller |       openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:56:26.821206 | controller |     - '    logger.go:42: 09:52:16 | watcher/1-deploy-with-defaults | + ''['' '''' == ''''
2026-03-14 09:56:26.821211 | controller |       '']'''
2026-03-14 09:56:26.821217 | controller |     - '    logger.go:42: 09:52:16 | watcher/1-deploy-with-defaults | + exit 0'
2026-03-14 09:56:26.821222 | controller |     - '    logger.go:42: 09:52:17 | watcher/1-deploy-with-defaults | running command:
2026-03-14 09:56:26.821227 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.821232 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.821237 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.821242 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:56:26.821247 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.821252 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.821257 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.821262 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.821269 | controller |       ]'
2026-03-14 09:56:26.821275 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.821280 | controller |       -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:56:26.821285 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.821290 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')"
2026-03-14 09:56:26.821295 | controller |       == 2 ]'
2026-03-14 09:56:26.821300 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.821305 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')"
2026-03-14 09:56:26.821310 | controller |       -ge 1 ]'
2026-03-14 09:56:26.821315 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.821320 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.821325 | controller |       == "" ]; then'
2026-03-14 09:56:26.821331 | controller |     - '            exit 0'
2026-03-14 09:56:26.821336 | controller |     - '        fi'
2026-03-14 09:56:26.821341 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.821346 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.821351 | controller |     - '        counter=0'
2026-03-14 09:56:26.821356 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.821361 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.821366 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.821371 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.821376 | controller |     - '            fi'
2026-03-14 09:56:26.821382 | controller |     - '        done'
2026-03-14 09:56:26.821387 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.821392 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.821397 | controller |     - '          exit 1'
2026-03-14 09:56:26.821402 | controller |     - '        else'
2026-03-14 09:56:26.821407 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.821412 | controller |     - '        fi'
2026-03-14 09:56:26.821418 | controller |     - '        ]'
2026-03-14 09:56:26.821423 | controller |     - '    logger.go:42: 09:52:17 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.821429 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:56:26.821434 | controller |     - '    logger.go:42: 09:52:17 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.821440 | controller |     - '    logger.go:42: 09:52:19 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.821445 | controller |     - '    logger.go:42: 09:52:19 | watcher/1-deploy-with-defaults | ++ grep watcher'
2026-03-14 09:56:26.821450 | controller |     - '    logger.go:42: 09:52:19 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.821455 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:56:26.821460 | controller |     - '    logger.go:42: 09:52:19 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}'''
2026-03-14 09:56:26.821465 | controller |     - '    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | + SERVICEID=e32ab6f2cb3c4ec69d47af38a3e246ce'
2026-03-14 09:56:26.821470 | controller |     - '    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.821475 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.821480 | controller |     - '    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | + ''['' e32ab6f2cb3c4ec69d47af38a3e246ce
2026-03-14 09:56:26.821485 | controller |       == e32ab6f2cb3c4ec69d47af38a3e246ce '']'''
2026-03-14 09:56:26.821492 | controller |     - '    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.821509 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.821515 | controller |     - '    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | + ''['' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.821522 | controller |       '']'''
2026-03-14 09:56:26.821527 | controller |     - '    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:56:26.821532 | controller |     - '    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.821537 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}'''
2026-03-14 09:56:26.821543 | controller |     - '    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1'
2026-03-14 09:56:26.821548 | controller |     - '    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.821553 | controller |     - '    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | ++ grep -c ''cafile
2026-03-14 09:56:26.821558 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'''
2026-03-14 09:56:26.821563 | controller |     - '    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:56:26.821568 | controller |     - '    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.821573 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:56:26.821578 | controller |     - '    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | + ''['' 2 == 2 '']'''
2026-03-14 09:56:26.821583 | controller |     - '    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:56:26.821588 | controller |     - '    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.821615 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:56:26.821623 | controller |     - '    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | ++ grep -c ''region_name
2026-03-14 09:56:26.821628 | controller |       = regionOne'''
2026-03-14 09:56:26.821633 | controller |     - '    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | + ''['' 9 -ge 1 '']'''
2026-03-14 09:56:26.821638 | controller |     - '    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | ++ oc get pods -n
2026-03-14 09:56:26.821643 | controller |       openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:56:26.821648 | controller |     - '    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | + ''['' '''' == ''''
2026-03-14 09:56:26.821654 | controller |       '']'''
2026-03-14 09:56:26.821659 | controller |     - '    logger.go:42: 09:52:22 | watcher/1-deploy-with-defaults | + exit 0'
2026-03-14 09:56:26.821664 | controller |     - '    logger.go:42: 09:52:23 | watcher/1-deploy-with-defaults | running command:
2026-03-14 09:56:26.821669 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.821674 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.821679 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.821685 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:56:26.821690 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.821695 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.821700 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.821706 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.821711 | controller |       ]'
2026-03-14 09:56:26.821716 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.821721 | controller |       -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:56:26.821729 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.821736 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')"
2026-03-14 09:56:26.821741 | controller |       == 2 ]'
2026-03-14 09:56:26.821746 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.821751 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')"
2026-03-14 09:56:26.821756 | controller |       -ge 1 ]'
2026-03-14 09:56:26.821761 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.821766 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.821772 | controller |       == "" ]; then'
2026-03-14 09:56:26.821777 | controller |     - '            exit 0'
2026-03-14 09:56:26.821782 | controller |     - '        fi'
2026-03-14 09:56:26.821787 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.821799 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.821805 | controller |     - '        counter=0'
2026-03-14 09:56:26.821810 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.821815 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.821820 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.821825 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.821831 | controller |     - '            fi'
2026-03-14 09:56:26.821836 | controller |     - '        done'
2026-03-14 09:56:26.821841 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.821858 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.821864 | controller |     - '          exit 1'
2026-03-14 09:56:26.821869 | controller |     - '        else'
2026-03-14 09:56:26.821874 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.821879 | controller |     - '        fi'
2026-03-14 09:56:26.821884 | controller |     - '        ]'
2026-03-14 09:56:26.821890 | controller |     - '    logger.go:42: 09:52:23 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.821895 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:56:26.821900 | controller |     - '    logger.go:42: 09:52:23 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.821905 | controller |     - '    logger.go:42: 09:52:25 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.821910 | controller |     - '    logger.go:42: 09:52:25 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.821915 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:56:26.821920 | controller |     - '    logger.go:42: 09:52:25 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}'''
2026-03-14 09:56:26.821925 | controller |     - '    logger.go:42: 09:52:25 | watcher/1-deploy-with-defaults | ++ grep watcher'
2026-03-14 09:56:26.821930 | controller |     - '    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults | + SERVICEID=e32ab6f2cb3c4ec69d47af38a3e246ce'
2026-03-14 09:56:26.821935 | controller |     - '    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.821940 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.821945 | controller |     - '    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults | + ''['' e32ab6f2cb3c4ec69d47af38a3e246ce
2026-03-14 09:56:26.821950 | controller |       == e32ab6f2cb3c4ec69d47af38a3e246ce '']'''
2026-03-14 09:56:26.821955 | controller |     - '    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.821961 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.821975 | controller |     - '    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults | + ''['' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.821984 | controller |       '']'''
2026-03-14 09:56:26.821990 | controller |     - '    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1'
2026-03-14 09:56:26.821995 | controller |     - '    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:56:26.822000 | controller |     - '    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.822005 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}'''
2026-03-14 09:56:26.822010 | controller |     - '    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.822015 | controller |     - '    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:56:26.822020 | controller |     - '    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.822025 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:56:26.822030 | controller |     - '    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults | ++ grep -c ''cafile
2026-03-14 09:56:26.822035 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'''
2026-03-14 09:56:26.822040 | controller |     - '    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults | + ''['' 2 == 2 '']'''
2026-03-14 09:56:26.822045 | controller |     - '    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:56:26.822050 | controller |     - '    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.822055 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:56:26.822060 | controller |     - '    logger.go:42: 09:52:28 | watcher/1-deploy-with-defaults | ++ grep -c ''region_name
2026-03-14 09:56:26.822065 | controller |       = regionOne'''
2026-03-14 09:56:26.822070 | controller |     - '    logger.go:42: 09:52:29 | watcher/1-deploy-with-defaults | + ''['' 9 -ge 1 '']'''
2026-03-14 09:56:26.822075 | controller |     - '    logger.go:42: 09:52:29 | watcher/1-deploy-with-defaults | ++ oc get pods -n
2026-03-14 09:56:26.822080 | controller |       openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:56:26.822085 | controller |     - '    logger.go:42: 09:52:29 | watcher/1-deploy-with-defaults | + ''['' '''' == ''''
2026-03-14 09:56:26.822090 | controller |       '']'''
2026-03-14 09:56:26.822095 | controller |     - '    logger.go:42: 09:52:29 | watcher/1-deploy-with-defaults | + exit 0'
2026-03-14 09:56:26.822100 | controller |     - '    logger.go:42: 09:52:30 | watcher/1-deploy-with-defaults | running command:
2026-03-14 09:56:26.822105 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.822111 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.822116 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.822128 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:56:26.822134 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.822139 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.822151 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.822159 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.822164 | controller |       ]'
2026-03-14 09:56:26.822169 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.822174 | controller |       -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:56:26.822179 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.822185 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')"
2026-03-14 09:56:26.822190 | controller |       == 2 ]'
2026-03-14 09:56:26.822197 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:56:26.822206 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')"
2026-03-14 09:56:26.822211 | controller |       -ge 1 ]'
2026-03-14 09:56:26.822218 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.822232 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.822237 | controller |       == "" ]; then'
2026-03-14 09:56:26.822244 | controller |     - '            exit 0'
2026-03-14 09:56:26.822254 | controller |     - '        fi'
2026-03-14 09:56:26.822259 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.822274 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.822279 | controller |     - '        counter=0'
2026-03-14 09:56:26.822284 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.822294 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.822299 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.822304 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.822309 | controller |     - '            fi'
2026-03-14 09:56:26.822314 | controller |     - '        done'
2026-03-14 09:56:26.822319 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.822324 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.822330 | controller |     - '          exit 1'
2026-03-14 09:56:26.822335 | controller |     - '        else'
2026-03-14 09:56:26.822340 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.822345 | controller |     - '        fi'
2026-03-14 09:56:26.822350 | controller |     - '        ]'
2026-03-14 09:56:26.822355 | controller |     - '    logger.go:42: 09:52:30 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.822360 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:56:26.822365 | controller |     - '    logger.go:42: 09:52:30 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.822370 | controller |     - '    logger.go:42: 09:52:32 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.822375 | controller |     - '    logger.go:42: 09:52:32 | watcher/1-deploy-with-defaults | ++ grep watcher'
2026-03-14 09:56:26.822381 | controller |     - '    logger.go:42: 09:52:32 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.822387 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:56:26.822392 | controller |     - '    logger.go:42: 09:52:32 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}'''
2026-03-14 09:56:26.822397 | controller |     - '    logger.go:42: 09:52:34 | watcher/1-deploy-with-defaults | + SERVICEID=e32ab6f2cb3c4ec69d47af38a3e246ce'
2026-03-14 09:56:26.822402 | controller |     - '    logger.go:42: 09:52:34 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.822407 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.822412 | controller |     - '    logger.go:42: 09:52:34 | watcher/1-deploy-with-defaults | + ''['' e32ab6f2cb3c4ec69d47af38a3e246ce
2026-03-14 09:56:26.822418 | controller |       == e32ab6f2cb3c4ec69d47af38a3e246ce '']'''
2026-03-14 09:56:26.822424 | controller |     - '    logger.go:42: 09:52:34 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.822429 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.822436 | controller |     - '    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | + ''['' -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.822449 | controller |       '']'''
2026-03-14 09:56:26.822455 | controller |     - '    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1'
2026-03-14 09:56:26.822460 | controller |     - '    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.822468 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}'''
2026-03-14 09:56:26.822473 | controller |     - '    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:56:26.822478 | controller |     - '    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']'''
2026-03-14 09:56:26.822483 | controller |     - '    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:56:26.822496 | controller |     - '    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | ++ grep -c ''cafile
2026-03-14 09:56:26.822502 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'''
2026-03-14 09:56:26.822507 | controller |     - '    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.822539 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:56:26.822545 | controller |     - '    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | + ''['' 2 == 2 '']'''
2026-03-14 09:56:26.822550 | controller |     - '    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:56:26.822555 | controller |     - '    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:56:26.822560 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:56:26.822565 | controller |     - '    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | ++ grep -c ''region_name
2026-03-14 09:56:26.822570 | controller |       = regionOne'''
2026-03-14 09:56:26.822575 | controller |     - '    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | + ''['' 9 -ge 1 '']'''
2026-03-14 09:56:26.822580 | controller |     - '    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | ++ oc get pods -n
2026-03-14 09:56:26.822585 | controller |       openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:56:26.822590 | controller |     - '    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | + ''['' '''' == ''''
2026-03-14 09:56:26.822614 | controller |       '']'''
2026-03-14 09:56:26.822620 | controller |     - '    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | + exit 0'
2026-03-14 09:56:26.822625 | controller |     - '    logger.go:42: 09:52:35 | watcher/1-deploy-with-defaults | test step completed
2026-03-14 09:56:26.822630 | controller |       1-deploy-with-defaults'
2026-03-14 09:56:26.822635 | controller |     - '    logger.go:42: 09:52:35 | watcher/2-cleanup-watcher | starting test step 2-cleanup-watcher'
2026-03-14 09:56:26.822640 | controller |     - '    logger.go:42: 09:52:35 | watcher/2-cleanup-watcher | test step completed 2-cleanup-watcher'
2026-03-14 09:56:26.822645 | controller |     - '    logger.go:42: 09:52:35 | watcher/3-precreate-mariadbaccount | starting test
2026-03-14 09:56:26.822650 | controller |       step 3-precreate-mariadbaccount'
2026-03-14 09:56:26.822655 | controller |     - '    logger.go:42: 09:52:35 | watcher/3-precreate-mariadbaccount | MariaDBAccount:watcher-kuttl-default/watcher-precreated
2026-03-14 09:56:26.822661 | controller |       created'
2026-03-14 09:56:26.822666 | controller |     - '    logger.go:42: 09:52:35 | watcher/3-precreate-mariadbaccount | test step completed
2026-03-14 09:56:26.822671 | controller |       3-precreate-mariadbaccount'
2026-03-14 09:56:26.822678 | controller |     - '    logger.go:42: 09:52:35 | watcher/4-deploy-with-precreated-account | starting
2026-03-14 09:56:26.822684 | controller |       test step 4-deploy-with-precreated-account'
2026-03-14 09:56:26.822689 | controller |     - '    logger.go:42: 09:52:35 | watcher/4-deploy-with-precreated-account | Secret:watcher-kuttl-default/custom-prometheus-config
2026-03-14 09:56:26.822695 | controller |       created'
2026-03-14 09:56:26.822707 | controller |     - '    logger.go:42: 09:52:35 | watcher/4-deploy-with-precreated-account | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:56:26.822712 | controller |       created'
2026-03-14 09:56:26.822717 | controller |     - '    logger.go:42: 09:52:35 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:56:26.822723 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.822728 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.822735 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.822740 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.822745 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.822750 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.822755 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:56:26.822760 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:56:26.822765 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:56:26.822770 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.822777 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:56:26.822783 | controller |       == 1 ]'
2026-03-14 09:56:26.822788 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.822794 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:56:26.822802 | controller |     - '        else'
2026-03-14 09:56:26.822807 | controller |     - '          exit 1'
2026-03-14 09:56:26.822812 | controller |     - '        fi'
2026-03-14 09:56:26.822821 | controller |     - '        ]'
2026-03-14 09:56:26.822826 | controller |     - '    logger.go:42: 09:52:35 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.822831 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.822836 | controller |     - '    logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:56:26.822841 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.822847 | controller |     - '    logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.822852 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.822857 | controller |     - '    logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.822862 | controller |       -1'
2026-03-14 09:56:26.822875 | controller |     - '    logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.822881 | controller |       -v ''^$'''
2026-03-14 09:56:26.822893 | controller |     - '    logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.822900 | controller |     - '    logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.822906 | controller |       -n watcher-kuttl-api-0 '']'''
2026-03-14 09:56:26.822910 | controller |     - '    logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.822915 | controller |       -c ''^# Global config'''
2026-03-14 09:56:26.822920 | controller |     - '    logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.822926 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf'
2026-03-14 09:56:26.822931 | controller |     - '    logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account | error:
2026-03-14 09:56:26.822937 | controller |       Internal error occurred: error executing command in container: container is not
2026-03-14 09:56:26.822942 | controller |       created or running'
2026-03-14 09:56:26.822947 | controller |     - '    logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account | ++ echo'
2026-03-14 09:56:26.822952 | controller |     - '    logger.go:42: 09:52:36 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.822958 | controller |       0 == 1 '']'''
2026-03-14 09:56:26.822969 | controller |     - '    logger.go:42: 09:52:37 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:56:26.822974 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.822980 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.822987 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.822992 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.822997 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.823002 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.823007 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:56:26.823012 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:56:26.823068 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:56:26.823077 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.823096 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:56:26.823102 | controller |       == 1 ]'
2026-03-14 09:56:26.823124 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.823130 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:56:26.823135 | controller |     - '        else'
2026-03-14 09:56:26.823141 | controller |     - '          exit 1'
2026-03-14 09:56:26.823152 | controller |     - '        fi'
2026-03-14 09:56:26.823158 | controller |     - '        ]'
2026-03-14 09:56:26.823163 | controller |     - '    logger.go:42: 09:52:37 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.823168 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.823174 | controller |     - '    logger.go:42: 09:52:38 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:56:26.823182 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.823188 | controller |     - '    logger.go:42: 09:52:38 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.823194 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.823199 | controller |     - '    logger.go:42: 09:52:38 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.823211 | controller |       -1'
2026-03-14 09:56:26.823221 | controller |     - '    logger.go:42: 09:52:38 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.823226 | controller |       -v ''^$'''
2026-03-14 09:56:26.823231 | controller |     - '    logger.go:42: 09:52:38 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:56:26.823236 | controller |     - '    logger.go:42: 09:52:39 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:56:26.823241 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.823246 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.823252 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.823259 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.823264 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.823270 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.823275 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:56:26.823280 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:56:26.823285 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:56:26.823290 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.823297 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:56:26.823303 | controller |       == 1 ]'
2026-03-14 09:56:26.823309 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.823318 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:56:26.823324 | controller |     - '        else'
2026-03-14 09:56:26.823329 | controller |     - '          exit 1'
2026-03-14 09:56:26.823334 | controller |     - '        fi'
2026-03-14 09:56:26.823355 | controller |     - '        ]'
2026-03-14 09:56:26.823362 | controller |     - '    logger.go:42: 09:52:39 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.823369 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.823374 | controller |     - '    logger.go:42: 09:52:39 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:56:26.823382 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.823387 | controller |     - '    logger.go:42: 09:52:39 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.823392 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.823397 | controller |     - '    logger.go:42: 09:52:39 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.823402 | controller |       -1'
2026-03-14 09:56:26.823408 | controller |     - '    logger.go:42: 09:52:39 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.823413 | controller |       -v ''^$'''
2026-03-14 09:56:26.823418 | controller |     - '    logger.go:42: 09:52:39 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:56:26.823423 | controller |     - '    logger.go:42: 09:52:40 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:56:26.823428 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.823434 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.823439 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.823444 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.823449 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.823454 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.823459 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:56:26.823465 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:56:26.823470 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:56:26.823475 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.823481 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:56:26.823486 | controller |       == 1 ]'
2026-03-14 09:56:26.823492 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.823497 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:56:26.823503 | controller |     - '        else'
2026-03-14 09:56:26.823508 | controller |     - '          exit 1'
2026-03-14 09:56:26.823513 | controller |     - '        fi'
2026-03-14 09:56:26.823518 | controller |     - '        ]'
2026-03-14 09:56:26.823523 | controller |     - '    logger.go:42: 09:52:40 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.823528 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.823533 | controller |     - '    logger.go:42: 09:52:40 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:56:26.823538 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.823543 | controller |     - '    logger.go:42: 09:52:40 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.823548 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.823553 | controller |     - '    logger.go:42: 09:52:40 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.823558 | controller |       -1'
2026-03-14 09:56:26.823571 | controller |     - '    logger.go:42: 09:52:40 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.823579 | controller |       -v ''^$'''
2026-03-14 09:56:26.823584 | controller |     - '    logger.go:42: 09:52:41 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:56:26.823589 | controller |     - '    logger.go:42: 09:52:42 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:56:26.823614 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.823633 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.823638 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.823643 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.823648 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.823654 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.823659 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:56:26.823664 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:56:26.823669 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:56:26.823681 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.823686 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:56:26.823691 | controller |       == 1 ]'
2026-03-14 09:56:26.823697 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.823702 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:56:26.823707 | controller |     - '        else'
2026-03-14 09:56:26.823712 | controller |     - '          exit 1'
2026-03-14 09:56:26.823717 | controller |     - '        fi'
2026-03-14 09:56:26.823722 | controller |     - '        ]'
2026-03-14 09:56:26.823736 | controller |     - '    logger.go:42: 09:52:42 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.823742 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.823747 | controller |     - '    logger.go:42: 09:52:42 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:56:26.823753 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.823758 | controller |     - '    logger.go:42: 09:52:42 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.823763 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.823768 | controller |     - '    logger.go:42: 09:52:42 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.823773 | controller |       -1'
2026-03-14 09:56:26.823778 | controller |     - '    logger.go:42: 09:52:42 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.823783 | controller |       -v ''^$'''
2026-03-14 09:56:26.823788 | controller |     - '    logger.go:42: 09:52:42 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:56:26.823798 | controller |     - '    logger.go:42: 09:52:43 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:56:26.823804 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.823809 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.823815 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.823820 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.823825 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.823830 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.823835 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:56:26.823840 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:56:26.823845 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:56:26.823851 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.823860 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:56:26.823865 | controller |       == 1 ]'
2026-03-14 09:56:26.823870 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.823875 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:56:26.823881 | controller |     - '        else'
2026-03-14 09:56:26.823886 | controller |     - '          exit 1'
2026-03-14 09:56:26.823891 | controller |     - '        fi'
2026-03-14 09:56:26.823896 | controller |     - '        ]'
2026-03-14 09:56:26.823901 | controller |     - '    logger.go:42: 09:52:43 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.823906 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.823911 | controller |     - '    logger.go:42: 09:52:43 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:56:26.823916 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.823925 | controller |     - '    logger.go:42: 09:52:43 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.823930 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.823936 | controller |     - '    logger.go:42: 09:52:43 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.823941 | controller |       -1'
2026-03-14 09:56:26.823946 | controller |     - '    logger.go:42: 09:52:43 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.823951 | controller |       -v ''^$'''
2026-03-14 09:56:26.823956 | controller |     - '    logger.go:42: 09:52:43 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:56:26.823961 | controller |     - '    logger.go:42: 09:52:45 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:56:26.823966 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.823971 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.823976 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.823981 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.823986 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.823991 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.823996 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:56:26.824002 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:56:26.824007 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:56:26.824012 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.824017 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:56:26.824022 | controller |       == 1 ]'
2026-03-14 09:56:26.824027 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.824032 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:56:26.824037 | controller |     - '        else'
2026-03-14 09:56:26.824042 | controller |     - '          exit 1'
2026-03-14 09:56:26.824047 | controller |     - '        fi'
2026-03-14 09:56:26.824052 | controller |     - '        ]'
2026-03-14 09:56:26.824057 | controller |     - '    logger.go:42: 09:52:45 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.824081 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.824096 | controller |     - '    logger.go:42: 09:52:45 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:56:26.824103 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.824110 | controller |     - '    logger.go:42: 09:52:45 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.824115 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.824120 | controller |     - '    logger.go:42: 09:52:45 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.824129 | controller |       -1'
2026-03-14 09:56:26.824135 | controller |     - '    logger.go:42: 09:52:45 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.824140 | controller |       -v ''^$'''
2026-03-14 09:56:26.824145 | controller |     - '    logger.go:42: 09:52:45 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:56:26.824166 | controller |     - '    logger.go:42: 09:52:46 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:56:26.824171 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.824176 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.824182 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.824187 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.824192 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.824197 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.824202 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:56:26.824214 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:56:26.824225 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:56:26.824230 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.824235 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:56:26.824240 | controller |       == 1 ]'
2026-03-14 09:56:26.824249 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.824255 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:56:26.824260 | controller |     - '        else'
2026-03-14 09:56:26.824265 | controller |     - '          exit 1'
2026-03-14 09:56:26.824270 | controller |     - '        fi'
2026-03-14 09:56:26.824275 | controller |     - '        ]'
2026-03-14 09:56:26.824280 | controller |     - '    logger.go:42: 09:52:46 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.824285 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.824290 | controller |     - '    logger.go:42: 09:52:46 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:56:26.824295 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.824300 | controller |     - '    logger.go:42: 09:52:46 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.824305 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.824310 | controller |     - '    logger.go:42: 09:52:46 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.824315 | controller |       -1'
2026-03-14 09:56:26.824320 | controller |     - '    logger.go:42: 09:52:46 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.824326 | controller |       -v ''^$'''
2026-03-14 09:56:26.824331 | controller |     - '    logger.go:42: 09:52:46 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:56:26.824336 | controller |     - '    logger.go:42: 09:52:47 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:56:26.824341 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.824346 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.824351 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.824356 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.824361 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.824370 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.824375 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:56:26.824381 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:56:26.824386 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:56:26.824391 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.824396 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:56:26.824401 | controller |       == 1 ]'
2026-03-14 09:56:26.824406 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.824412 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:56:26.824417 | controller |     - '        else'
2026-03-14 09:56:26.824422 | controller |     - '          exit 1'
2026-03-14 09:56:26.824427 | controller |     - '        fi'
2026-03-14 09:56:26.824432 | controller |     - '        ]'
2026-03-14 09:56:26.824437 | controller |     - '    logger.go:42: 09:52:47 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.824442 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.824454 | controller |     - '    logger.go:42: 09:52:47 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:56:26.824459 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.824464 | controller |     - '    logger.go:42: 09:52:47 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.824472 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.824477 | controller |     - '    logger.go:42: 09:52:47 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.824482 | controller |       -1'
2026-03-14 09:56:26.824487 | controller |     - '    logger.go:42: 09:52:47 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.824492 | controller |       -v ''^$'''
2026-03-14 09:56:26.824497 | controller |     - '    logger.go:42: 09:52:48 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:56:26.824503 | controller |     - '    logger.go:42: 09:52:49 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:56:26.824508 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.824513 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.824518 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.824523 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.824529 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.824554 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.824559 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:56:26.824567 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:56:26.824572 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:56:26.824577 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.824583 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:56:26.824588 | controller |       == 1 ]'
2026-03-14 09:56:26.824593 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.824611 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:56:26.824617 | controller |     - '        else'
2026-03-14 09:56:26.824622 | controller |     - '          exit 1'
2026-03-14 09:56:26.824627 | controller |     - '        fi'
2026-03-14 09:56:26.824635 | controller |     - '        ]'
2026-03-14 09:56:26.824640 | controller |     - '    logger.go:42: 09:52:49 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.824646 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.824651 | controller |     - '    logger.go:42: 09:52:49 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:56:26.824656 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.824661 | controller |     - '    logger.go:42: 09:52:49 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.824670 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.824676 | controller |     - '    logger.go:42: 09:52:49 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.824681 | controller |       -1'
2026-03-14 09:56:26.824686 | controller |     - '    logger.go:42: 09:52:49 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.824691 | controller |       -v ''^$'''
2026-03-14 09:56:26.824696 | controller |     - '    logger.go:42: 09:52:49 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:56:26.824701 | controller |     - '    logger.go:42: 09:52:50 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:56:26.824706 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.824711 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.824716 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.824721 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.824726 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.824731 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.824736 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:56:26.824741 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:56:26.824746 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:56:26.824751 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.824756 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:56:26.824761 | controller |       == 1 ]'
2026-03-14 09:56:26.824766 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.824778 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:56:26.824783 | controller |     - '        else'
2026-03-14 09:56:26.824788 | controller |     - '          exit 1'
2026-03-14 09:56:26.824793 | controller |     - '        fi'
2026-03-14 09:56:26.824798 | controller |     - '        ]'
2026-03-14 09:56:26.824803 | controller |     - '    logger.go:42: 09:52:50 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.824808 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.824813 | controller |     - '    logger.go:42: 09:52:50 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:56:26.824826 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.824831 | controller |     - '    logger.go:42: 09:52:50 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.824836 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.824842 | controller |     - '    logger.go:42: 09:52:50 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.824847 | controller |       -1'
2026-03-14 09:56:26.824854 | controller |     - '    logger.go:42: 09:52:50 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.824860 | controller |       -v ''^$'''
2026-03-14 09:56:26.824865 | controller |     - '    logger.go:42: 09:52:50 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:56:26.824871 | controller |     - '    logger.go:42: 09:52:51 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:56:26.824880 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.824886 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.824892 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.824898 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.824903 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.824909 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.824914 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:56:26.824919 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:56:26.824924 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:56:26.824929 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.824936 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:56:26.824942 | controller |       == 1 ]'
2026-03-14 09:56:26.824947 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.824952 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:56:26.824957 | controller |     - '        else'
2026-03-14 09:56:26.824962 | controller |     - '          exit 1'
2026-03-14 09:56:26.824967 | controller |     - '        fi'
2026-03-14 09:56:26.824974 | controller |     - '        ]'
2026-03-14 09:56:26.824979 | controller |     - '    logger.go:42: 09:52:51 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.824984 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.824989 | controller |     - '    logger.go:42: 09:52:52 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:56:26.824994 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.824999 | controller |     - '    logger.go:42: 09:52:52 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.825004 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.825011 | controller |     - '    logger.go:42: 09:52:52 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.825016 | controller |       -1'
2026-03-14 09:56:26.825021 | controller |     - '    logger.go:42: 09:52:52 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.825026 | controller |       -v ''^$'''
2026-03-14 09:56:26.825031 | controller |     - '    logger.go:42: 09:52:52 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.825036 | controller |     - '    logger.go:42: 09:52:52 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.825041 | controller |       -n watcher-kuttl-api-0 '']'''
2026-03-14 09:56:26.825046 | controller |     - '    logger.go:42: 09:52:52 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.825051 | controller |       -c ''^# Global config'''
2026-03-14 09:56:26.825057 | controller |     - '    logger.go:42: 09:52:52 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.825062 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf'
2026-03-14 09:56:26.825067 | controller |     - '    logger.go:42: 09:52:52 | watcher/4-deploy-with-precreated-account | error:
2026-03-14 09:56:26.825072 | controller |       unable to upgrade connection: container not found ("watcher-api")'
2026-03-14 09:56:26.825077 | controller |     - '    logger.go:42: 09:52:52 | watcher/4-deploy-with-precreated-account | ++ echo'
2026-03-14 09:56:26.825082 | controller |     - '    logger.go:42: 09:52:52 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.825088 | controller |       0 == 1 '']'''
2026-03-14 09:56:26.825095 | controller |     - '    logger.go:42: 09:52:53 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:56:26.825100 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.825108 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.825113 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.825119 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.825124 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.825129 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.825134 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:56:26.825139 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:56:26.825144 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:56:26.825161 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.825167 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:56:26.825172 | controller |       == 1 ]'
2026-03-14 09:56:26.825177 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.825182 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:56:26.825187 | controller |     - '        else'
2026-03-14 09:56:26.825192 | controller |     - '          exit 1'
2026-03-14 09:56:26.825197 | controller |     - '        fi'
2026-03-14 09:56:26.825202 | controller |     - '        ]'
2026-03-14 09:56:26.825207 | controller |     - '    logger.go:42: 09:52:53 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.825212 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.825217 | controller |     - '    logger.go:42: 09:52:53 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:56:26.825222 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.825227 | controller |     - '    logger.go:42: 09:52:53 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.825232 | controller |       -1'
2026-03-14 09:56:26.825237 | controller |     - '    logger.go:42: 09:52:53 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.825243 | controller |       -v ''^$'''
2026-03-14 09:56:26.825248 | controller |     - '    logger.go:42: 09:52:53 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.825253 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.825258 | controller |     - '    logger.go:42: 09:52:53 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.825263 | controller |     - '    logger.go:42: 09:52:53 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.825270 | controller |       -n watcher-kuttl-api-0 '']'''
2026-03-14 09:56:26.825275 | controller |     - '    logger.go:42: 09:52:53 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.825280 | controller |       -c ''^# Global config'''
2026-03-14 09:56:26.825285 | controller |     - '    logger.go:42: 09:52:53 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.825291 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf'
2026-03-14 09:56:26.825296 | controller |     - '    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.825301 | controller |       ''#'' Global config'
2026-03-14 09:56:26.825308 | controller |     - '    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.825313 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.825318 | controller |     - '    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.825324 | controller |       -c ''^# Service config'''
2026-03-14 09:56:26.825329 | controller |     - '    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.825334 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf'
2026-03-14 09:56:26.825342 | controller |     - '    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.825348 | controller |       ''#'' Service config'
2026-03-14 09:56:26.825353 | controller |     - '    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.825358 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.825363 | controller |     - '    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.825368 | controller |       -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'''
2026-03-14 09:56:26.825373 | controller |     - '    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.825378 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.825384 | controller |     - '    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.825389 | controller |       ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.825394 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.825399 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.825404 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_test:bcdc991c7f059e29ee45381194286841@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.825409 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.825413 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.825418 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.825423 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.825428 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.825445 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.825450 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.825455 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.825460 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.825465 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.825470 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.825475 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.825480 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.825485 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.825490 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.825495 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.825500 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.825505 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.825510 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.825517 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]''
2026-03-14 09:56:26.825522 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.825527 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.825532 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.825537 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.825542 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.825547 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.825552 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.825557 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.825562 | controller |     - '    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.825567 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.825573 | controller |     - '    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.825586 | controller |       -czPo ''TimeOut 80'''
2026-03-14 09:56:26.825591 | controller |     - '    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.825614 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf'
2026-03-14 09:56:26.825619 | controller |     - '    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.825625 | controller |       ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:56:26.825630 | controller |       ''*:9322>'' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:56:26.825634 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.825639 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.825644 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.825649 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:56:26.825654 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:56:26.825659 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:56:26.825667 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:56:26.825672 | controller |       internal display-name=internal group=watcher processes=2 threads=1 user=watcher
2026-03-14 09:56:26.825677 | controller |       WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>''
2026-03-14 09:56:26.825682 | controller |       ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:56:26.825687 | controller |       ''*:9322>'' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:56:26.825693 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.825699 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.825716 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.825722 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:56:26.825728 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:56:26.825734 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:56:26.825742 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:56:26.825748 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:56:26.825753 | controller |       public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>'''
2026-03-14 09:56:26.825759 | controller |     - '    logger.go:42: 09:52:54 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.825764 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.825770 | controller |     - '    logger.go:42: 09:52:55 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:56:26.825775 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.825779 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.825784 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.825788 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.825792 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.825796 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.825800 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:56:26.825805 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:56:26.825809 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:56:26.825813 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.825817 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:56:26.825822 | controller |       == 1 ]'
2026-03-14 09:56:26.825827 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.825832 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:56:26.825838 | controller |     - '        else'
2026-03-14 09:56:26.825844 | controller |     - '          exit 1'
2026-03-14 09:56:26.825849 | controller |     - '        fi'
2026-03-14 09:56:26.825855 | controller |     - '        ]'
2026-03-14 09:56:26.825860 | controller |     - '    logger.go:42: 09:52:55 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.825866 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.825871 | controller |     - '    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:56:26.825877 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.825882 | controller |     - '    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.825888 | controller |       -1'
2026-03-14 09:56:26.825893 | controller |     - '    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.825899 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.825904 | controller |     - '    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.825909 | controller |       -v ''^$'''
2026-03-14 09:56:26.825915 | controller |     - '    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.825920 | controller |     - '    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.825926 | controller |       -n watcher-kuttl-api-0 '']'''
2026-03-14 09:56:26.825931 | controller |     - '    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.825936 | controller |       -c ''^# Global config'''
2026-03-14 09:56:26.825940 | controller |     - '    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.825944 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf'
2026-03-14 09:56:26.825948 | controller |     - '    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.825956 | controller |       ''#'' Global config'
2026-03-14 09:56:26.825961 | controller |     - '    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.825965 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.825969 | controller |     - '    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.825973 | controller |       -c ''^# Service config'''
2026-03-14 09:56:26.825979 | controller |     - '    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.825985 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf'
2026-03-14 09:56:26.825990 | controller |     - '    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.825996 | controller |       ''#'' Service config'
2026-03-14 09:56:26.826001 | controller |     - '    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.826007 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.826013 | controller |     - '    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.826029 | controller |       -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'''
2026-03-14 09:56:26.826036 | controller |     - '    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.826041 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.826047 | controller |     - '    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.826053 | controller |       ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.826059 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.826065 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.826071 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_test:bcdc991c7f059e29ee45381194286841@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.826076 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.826082 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.826088 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.826094 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.826100 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.826106 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.826111 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.826120 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.826126 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.826132 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.826137 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.826143 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.826149 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.826154 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.826162 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.826168 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.826174 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.826179 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.826185 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.826190 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]''
2026-03-14 09:56:26.826196 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.826202 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.826207 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.826213 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.826218 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.826224 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.826229 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.826235 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.826241 | controller |     - '    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.826247 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.826252 | controller |     - '    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.826258 | controller |       -czPo ''TimeOut 80'''
2026-03-14 09:56:26.826264 | controller |     - '    logger.go:42: 09:52:56 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.826269 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf'
2026-03-14 09:56:26.826282 | controller |     - '    logger.go:42: 09:52:57 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.826288 | controller |       ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:56:26.826294 | controller |       ''*:9322>'' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:56:26.826300 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.826305 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.826311 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.826317 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:56:26.826322 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:56:26.826328 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:56:26.826333 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:56:26.826340 | controller |       internal display-name=internal group=watcher processes=2 threads=1 user=watcher
2026-03-14 09:56:26.826347 | controller |       WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>''
2026-03-14 09:56:26.826353 | controller |       ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:56:26.826359 | controller |       ''*:9322>'' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:56:26.826369 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.826375 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.826381 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.826386 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:56:26.826394 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:56:26.826400 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:56:26.826406 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:56:26.826411 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:56:26.826417 | controller |       public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>'''
2026-03-14 09:56:26.826423 | controller |     - '    logger.go:42: 09:52:57 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.826428 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.826434 | controller |     - '    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:56:26.826439 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.826445 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.826451 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.826456 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.826462 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.826468 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.826473 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:56:26.826479 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:56:26.826485 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:56:26.826490 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.826496 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:56:26.826502 | controller |       == 1 ]'
2026-03-14 09:56:26.826510 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.826516 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:56:26.826521 | controller |     - '        else'
2026-03-14 09:56:26.826527 | controller |     - '          exit 1'
2026-03-14 09:56:26.826533 | controller |     - '        fi'
2026-03-14 09:56:26.826538 | controller |     - '        ]'
2026-03-14 09:56:26.826544 | controller |     - '    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.826550 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.826555 | controller |     - '    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:56:26.826561 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.826567 | controller |     - '    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.826572 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.826578 | controller |     - '    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.826583 | controller |       -v ''^$'''
2026-03-14 09:56:26.826589 | controller |     - '    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.826612 | controller |       -1'
2026-03-14 09:56:26.826629 | controller |     - '    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.826638 | controller |     - '    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.826644 | controller |       -n watcher-kuttl-api-0 '']'''
2026-03-14 09:56:26.826650 | controller |     - '    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.826655 | controller |       -c ''^# Global config'''
2026-03-14 09:56:26.826661 | controller |     - '    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.826667 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf'
2026-03-14 09:56:26.826672 | controller |     - '    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.826678 | controller |       ''#'' Global config'
2026-03-14 09:56:26.826683 | controller |     - '    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.826689 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.826695 | controller |     - '    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.826700 | controller |       -c ''^# Service config'''
2026-03-14 09:56:26.826706 | controller |     - '    logger.go:42: 09:52:58 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.826712 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf'
2026-03-14 09:56:26.826717 | controller |     - '    logger.go:42: 09:52:59 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.826723 | controller |       ''#'' Service config'
2026-03-14 09:56:26.826729 | controller |     - '    logger.go:42: 09:52:59 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.826734 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.826740 | controller |     - '    logger.go:42: 09:52:59 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.826745 | controller |       -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'''
2026-03-14 09:56:26.826751 | controller |     - '    logger.go:42: 09:52:59 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.826757 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.826763 | controller |     - '    logger.go:42: 09:52:59 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.826768 | controller |       ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.826777 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.826783 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.826788 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_test:bcdc991c7f059e29ee45381194286841@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.826794 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.826800 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.826805 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.826811 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.826816 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.826822 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.826828 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.826833 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.826841 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.826847 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.826853 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.826858 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.826864 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.826869 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.826875 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.826881 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.826886 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.826900 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.826906 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.826912 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]''
2026-03-14 09:56:26.826918 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.826924 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.826929 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.826935 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.826941 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.826946 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.826952 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.826958 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.826963 | controller |     - '    logger.go:42: 09:52:59 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.826969 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.826974 | controller |     - '    logger.go:42: 09:52:59 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.826980 | controller |       -czPo ''TimeOut 80'''
2026-03-14 09:56:26.826986 | controller |     - '    logger.go:42: 09:52:59 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.826991 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf'
2026-03-14 09:56:26.826997 | controller |     - '    logger.go:42: 09:52:59 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.827003 | controller |       ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:56:26.827009 | controller |       ''*:9322>'' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:56:26.827014 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.827020 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.827025 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.827031 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:56:26.827036 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:56:26.827045 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:56:26.827063 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:56:26.827069 | controller |       internal display-name=internal group=watcher processes=2 threads=1 user=watcher
2026-03-14 09:56:26.827075 | controller |       WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>''
2026-03-14 09:56:26.827080 | controller |       ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:56:26.827086 | controller |       ''*:9322>'' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:56:26.827092 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.827097 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.827103 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.827108 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:56:26.827114 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:56:26.827120 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:56:26.827125 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:56:26.827131 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:56:26.827139 | controller |       public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>'''
2026-03-14 09:56:26.827146 | controller |     - '    logger.go:42: 09:52:59 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.827151 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.827157 | controller |     - '    logger.go:42: 09:53:00 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:56:26.827163 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.827169 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.827175 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.827181 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.827186 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.827192 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.827207 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:56:26.827213 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:56:26.827219 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:56:26.827225 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.827233 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:56:26.827239 | controller |       == 1 ]'
2026-03-14 09:56:26.827245 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.827251 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:56:26.827256 | controller |     - '        else'
2026-03-14 09:56:26.827262 | controller |     - '          exit 1'
2026-03-14 09:56:26.827268 | controller |     - '        fi'
2026-03-14 09:56:26.827274 | controller |     - '        ]'
2026-03-14 09:56:26.827279 | controller |     - '    logger.go:42: 09:53:00 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.827285 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.827291 | controller |     - '    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:56:26.827300 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.827306 | controller |     - '    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.827311 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.827317 | controller |     - '    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.827323 | controller |       -1'
2026-03-14 09:56:26.827329 | controller |     - '    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.827335 | controller |       -v ''^$'''
2026-03-14 09:56:26.827341 | controller |     - '    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.827346 | controller |     - '    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.827352 | controller |       -n watcher-kuttl-api-0 '']'''
2026-03-14 09:56:26.827358 | controller |     - '    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.827364 | controller |       -c ''^# Global config'''
2026-03-14 09:56:26.827370 | controller |     - '    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.827375 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf'
2026-03-14 09:56:26.827381 | controller |     - '    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.827387 | controller |       ''#'' Global config'
2026-03-14 09:56:26.827393 | controller |     - '    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.827399 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.827405 | controller |     - '    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.827410 | controller |       -c ''^# Service config'''
2026-03-14 09:56:26.827416 | controller |     - '    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.827422 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf'
2026-03-14 09:56:26.827427 | controller |     - '    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.827433 | controller |       ''#'' Service config'
2026-03-14 09:56:26.827439 | controller |     - '    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.827445 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.827451 | controller |     - '    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.827457 | controller |       -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'''
2026-03-14 09:56:26.827462 | controller |     - '    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.827468 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.827474 | controller |     - '    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.827480 | controller |       ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.827486 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.827492 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.827498 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_test:bcdc991c7f059e29ee45381194286841@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.827506 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.827512 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.827529 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.827535 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.827541 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.827547 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.827553 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.827559 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.827565 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.827570 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.827576 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.827582 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.827588 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.827611 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.827620 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.827626 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.827632 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.827637 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.827643 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.827649 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]''
2026-03-14 09:56:26.827654 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.827660 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.827666 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.827672 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.827677 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.827683 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.827688 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.827694 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.827699 | controller |     - '    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.827705 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.827710 | controller |     - '    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.827716 | controller |       -czPo ''TimeOut 80'''
2026-03-14 09:56:26.827721 | controller |     - '    logger.go:42: 09:53:01 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.827727 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf'
2026-03-14 09:56:26.827731 | controller |     - '    logger.go:42: 09:53:02 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.827736 | controller |       ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:56:26.827742 | controller |       ''*:9322>'' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:56:26.827747 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.827751 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.827755 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.827759 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:56:26.827763 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:56:26.827768 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:56:26.827772 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:56:26.827776 | controller |       internal display-name=internal group=watcher processes=2 threads=1 user=watcher
2026-03-14 09:56:26.827780 | controller |       WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>''
2026-03-14 09:56:26.827785 | controller |       ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:56:26.827796 | controller |       ''*:9322>'' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:56:26.827801 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.827805 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.827809 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.827814 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:56:26.827818 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:56:26.827822 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:56:26.827827 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:56:26.827831 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:56:26.827835 | controller |       public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>'''
2026-03-14 09:56:26.827839 | controller |     - '    logger.go:42: 09:53:02 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.827843 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.827858 | controller |     - '    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:56:26.827862 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.827867 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.827871 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.827875 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.827879 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.827884 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.827888 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:56:26.827892 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:56:26.827896 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:56:26.827900 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.827905 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:56:26.827909 | controller |       == 1 ]'
2026-03-14 09:56:26.827915 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.827919 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:56:26.827923 | controller |     - '        else'
2026-03-14 09:56:26.827928 | controller |     - '          exit 1'
2026-03-14 09:56:26.827934 | controller |     - '        fi'
2026-03-14 09:56:26.827940 | controller |     - '        ]'
2026-03-14 09:56:26.827944 | controller |     - '    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.827948 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.827953 | controller |     - '    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:56:26.827957 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.827961 | controller |     - '    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.827965 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.827970 | controller |     - '    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.827974 | controller |       -1'
2026-03-14 09:56:26.827978 | controller |     - '    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.827982 | controller |       -v ''^$'''
2026-03-14 09:56:26.827986 | controller |     - '    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.827991 | controller |     - '    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.827995 | controller |       -n watcher-kuttl-api-0 '']'''
2026-03-14 09:56:26.827999 | controller |     - '    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.828003 | controller |       -c ''^# Global config'''
2026-03-14 09:56:26.828008 | controller |     - '    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.828012 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf'
2026-03-14 09:56:26.828016 | controller |     - '    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.828020 | controller |       ''#'' Global config'
2026-03-14 09:56:26.828025 | controller |     - '    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.828029 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.828033 | controller |     - '    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.828037 | controller |       -c ''^# Service config'''
2026-03-14 09:56:26.828042 | controller |     - '    logger.go:42: 09:53:03 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.828046 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf'
2026-03-14 09:56:26.828050 | controller |     - '    logger.go:42: 09:53:04 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.828054 | controller |       ''#'' Service config'
2026-03-14 09:56:26.828065 | controller |     - '    logger.go:42: 09:53:04 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.828069 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.828073 | controller |     - '    logger.go:42: 09:53:04 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.828078 | controller |       -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'''
2026-03-14 09:56:26.828082 | controller |     - '    logger.go:42: 09:53:04 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.828086 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.828091 | controller |     - '    logger.go:42: 09:53:04 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.828095 | controller |       ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.828101 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.828105 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.828109 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_test:bcdc991c7f059e29ee45381194286841@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.828114 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.828118 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.828122 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.828126 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.828131 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.828135 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.828139 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.828143 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.828147 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.828152 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.828156 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.828160 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.828164 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.828169 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.828173 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.828177 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.828181 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.828187 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.828193 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.828197 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]''
2026-03-14 09:56:26.828201 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.828206 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.828210 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.828214 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.828220 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.828224 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.828228 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.828232 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.828237 | controller |     - '    logger.go:42: 09:53:04 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.828243 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.828247 | controller |     - '    logger.go:42: 09:53:04 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.828257 | controller |       -czPo ''TimeOut 80'''
2026-03-14 09:56:26.828262 | controller |     - '    logger.go:42: 09:53:04 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.828266 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf'
2026-03-14 09:56:26.828270 | controller |     - '    logger.go:42: 09:53:04 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.828275 | controller |       ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:56:26.828279 | controller |       ''*:9322>'' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:56:26.828283 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.828287 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.828291 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.828296 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:56:26.828300 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:56:26.828304 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:56:26.828308 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:56:26.828312 | controller |       internal display-name=internal group=watcher processes=2 threads=1 user=watcher
2026-03-14 09:56:26.828317 | controller |       WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>''
2026-03-14 09:56:26.828321 | controller |       ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:56:26.828325 | controller |       ''*:9322>'' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:56:26.828329 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.828333 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.828338 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.828342 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:56:26.828346 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:56:26.828350 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:56:26.828354 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:56:26.828359 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:56:26.828363 | controller |       public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>'''
2026-03-14 09:56:26.828367 | controller |     - '    logger.go:42: 09:53:04 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.828371 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.828375 | controller |     - '    logger.go:42: 09:53:05 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:56:26.828380 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.828384 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.828388 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.828392 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.828399 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.828403 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.828409 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:56:26.828415 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:56:26.828419 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:56:26.828423 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.828428 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:56:26.828432 | controller |       == 1 ]'
2026-03-14 09:56:26.828436 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.828441 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:56:26.828445 | controller |     - '        else'
2026-03-14 09:56:26.828449 | controller |     - '          exit 1'
2026-03-14 09:56:26.828453 | controller |     - '        fi'
2026-03-14 09:56:26.828458 | controller |     - '        ]'
2026-03-14 09:56:26.828463 | controller |     - '    logger.go:42: 09:53:05 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.828468 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.828472 | controller |     - '    logger.go:42: 09:53:05 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:56:26.828476 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.828481 | controller |     - '    logger.go:42: 09:53:05 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.828491 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.828495 | controller |     - '    logger.go:42: 09:53:05 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.828500 | controller |       -1'
2026-03-14 09:56:26.828504 | controller |     - '    logger.go:42: 09:53:05 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.828508 | controller |       -v ''^$'''
2026-03-14 09:56:26.828512 | controller |     - '    logger.go:42: 09:53:05 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.828517 | controller |     - '    logger.go:42: 09:53:05 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.828521 | controller |       -n watcher-kuttl-api-0 '']'''
2026-03-14 09:56:26.828525 | controller |     - '    logger.go:42: 09:53:05 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.828530 | controller |       -c ''^# Global config'''
2026-03-14 09:56:26.828534 | controller |     - '    logger.go:42: 09:53:05 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.828538 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf'
2026-03-14 09:56:26.828542 | controller |     - '    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.828546 | controller |       ''#'' Global config'
2026-03-14 09:56:26.828551 | controller |     - '    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.828555 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.828559 | controller |     - '    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.828563 | controller |       -c ''^# Service config'''
2026-03-14 09:56:26.828568 | controller |     - '    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.828572 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf'
2026-03-14 09:56:26.828576 | controller |     - '    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.828580 | controller |       ''#'' Service config'
2026-03-14 09:56:26.828585 | controller |     - '    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.828589 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.828605 | controller |     - '    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.828611 | controller |       -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'''
2026-03-14 09:56:26.828633 | controller |     - '    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.828638 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.828642 | controller |     - '    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.828646 | controller |       ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.828650 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.828655 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.828661 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_test:bcdc991c7f059e29ee45381194286841@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.828665 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.828669 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.828674 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.828678 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.828682 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.828686 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.828691 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.828695 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.828699 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.828703 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.828707 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.828712 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.828716 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.828720 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.828731 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.828735 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.828740 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.828744 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.828748 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.828753 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]''
2026-03-14 09:56:26.828759 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.828763 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.828767 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.828773 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.828778 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.828782 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.828786 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.828790 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.828794 | controller |     - '    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.828799 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.828803 | controller |     - '    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.828807 | controller |       -czPo ''TimeOut 80'''
2026-03-14 09:56:26.828811 | controller |     - '    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.828816 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf'
2026-03-14 09:56:26.828820 | controller |     - '    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.828824 | controller |       ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:56:26.828828 | controller |       ''*:9322>'' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:56:26.828833 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.828837 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.828841 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.828845 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:56:26.828850 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:56:26.828854 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:56:26.828858 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:56:26.828862 | controller |       internal display-name=internal group=watcher processes=2 threads=1 user=watcher
2026-03-14 09:56:26.828867 | controller |       WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>''
2026-03-14 09:56:26.828872 | controller |       ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:56:26.828877 | controller |       ''*:9322>'' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:56:26.828881 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.828885 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.828890 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.828894 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:56:26.828898 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:56:26.828902 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:56:26.828906 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:56:26.828911 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:56:26.828915 | controller |       public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>'''
2026-03-14 09:56:26.828919 | controller |     - '    logger.go:42: 09:53:06 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.828925 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.828929 | controller |     - '    logger.go:42: 09:53:07 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:56:26.828933 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.828938 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:56:26.828948 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:56:26.828952 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:56:26.828957 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:56:26.828961 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:56:26.828965 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:56:26.828969 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:56:26.828974 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:56:26.828978 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:56:26.828982 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:56:26.828986 | controller |       == 1 ]'
2026-03-14 09:56:26.828991 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:56:26.828995 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:56:26.828999 | controller |     - '        else'
2026-03-14 09:56:26.829005 | controller |     - '          exit 1'
2026-03-14 09:56:26.829010 | controller |     - '        fi'
2026-03-14 09:56:26.829014 | controller |     - '        ]'
2026-03-14 09:56:26.829018 | controller |     - '    logger.go:42: 09:53:07 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:56:26.829022 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.829027 | controller |     - '    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:56:26.829031 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:56:26.829035 | controller |     - '    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:56:26.829039 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:56:26.829044 | controller |     - '    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:56:26.829048 | controller |       -1'
2026-03-14 09:56:26.829052 | controller |     - '    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.829056 | controller |       -v ''^$'''
2026-03-14 09:56:26.829061 | controller |     - '    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:56:26.829065 | controller |     - '    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.829069 | controller |       -n watcher-kuttl-api-0 '']'''
2026-03-14 09:56:26.829073 | controller |     - '    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.829078 | controller |       -c ''^# Global config'''
2026-03-14 09:56:26.829083 | controller |     - '    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.829088 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf'
2026-03-14 09:56:26.829092 | controller |     - '    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.829096 | controller |       ''#'' Global config'
2026-03-14 09:56:26.829100 | controller |     - '    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.829105 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.829109 | controller |     - '    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.829113 | controller |       -c ''^# Service config'''
2026-03-14 09:56:26.829119 | controller |     - '    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.829123 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf'
2026-03-14 09:56:26.829128 | controller |     - '    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.829132 | controller |       ''#'' Service config'
2026-03-14 09:56:26.829136 | controller |     - '    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.829140 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.829145 | controller |     - '    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.829149 | controller |       -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'''
2026-03-14 09:56:26.829153 | controller |     - '    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.829157 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:56:26.829162 | controller |     - '    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.829166 | controller |       ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_3_NIVbWEs9R6eerLDXT:PL8rXcmhI8Cs1rnHLUacbMSr45jxQDyv@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:56:26.829170 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:56:26.829174 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:56:26.829185 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_test:bcdc991c7f059e29ee45381194286841@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:56:26.829189 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:56:26.829194 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:56:26.829198 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.829202 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:56:26.829206 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:56:26.829211 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:56:26.829215 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.829219 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.829223 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:56:26.829228 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:56:26.829232 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:56:26.829236 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:56:26.829241 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:56:26.829245 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:56:26.829249 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:56:26.829253 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:56:26.829258 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:56:26.829263 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:56:26.829268 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:56:26.829272 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]''
2026-03-14 09:56:26.829276 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:56:26.829280 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:56:26.829285 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:56:26.829289 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:56:26.829293 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:56:26.829297 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:56:26.829301 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:56:26.829306 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:56:26.829310 | controller |     - '    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.829314 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.829320 | controller |     - '    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:56:26.829325 | controller |       -czPo ''TimeOut 80'''
2026-03-14 09:56:26.829329 | controller |     - '    logger.go:42: 09:53:08 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:56:26.829333 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf'
2026-03-14 09:56:26.829337 | controller |     - '    logger.go:42: 09:53:09 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:56:26.829342 | controller |       ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:56:26.829346 | controller |       ''*:9322>'' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:56:26.829350 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.829354 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.829358 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.829363 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:56:26.829367 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:56:26.829371 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:56:26.829375 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:56:26.829385 | controller |       internal display-name=internal group=watcher processes=2 threads=1 user=watcher
2026-03-14 09:56:26.829390 | controller |       WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>''
2026-03-14 09:56:26.829394 | controller |       ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:56:26.829398 | controller |       ''*:9322>'' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:56:26.829402 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:56:26.829407 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:56:26.829411 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:56:26.829415 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:56:26.829421 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:56:26.829425 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:56:26.829429 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:56:26.829434 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:56:26.829438 | controller |       public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>'''
2026-03-14 09:56:26.829442 | controller |     - '    logger.go:42: 09:53:09 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:56:26.829446 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.829451 | controller |     - '    logger.go:42: 09:53:09 | watcher/4-deploy-with-precreated-account | test step
2026-03-14 09:56:26.829455 | controller |       completed 4-deploy-with-precreated-account'
2026-03-14 09:56:26.829459 | controller |     - '    logger.go:42: 09:53:09 | watcher/5-cleanup-watcher | starting test step 5-cleanup-watcher'
2026-03-14 09:56:26.829463 | controller |     - '    logger.go:42: 09:53:09 | watcher/5-cleanup-watcher | running command: [sh -c
2026-03-14 09:56:26.829467 | controller |       set -ex'
2026-03-14 09:56:26.829472 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.829476 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:56:26.829481 | controller |     - '        ]'
2026-03-14 09:56:26.829485 | controller |     - '    logger.go:42: 09:53:09 | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.829489 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:56:26.829493 | controller |     - '    logger.go:42: 09:53:09 | watcher/5-cleanup-watcher | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.829498 | controller |     - '    logger.go:42: 09:53:12 | watcher/5-cleanup-watcher | + ''['' 1 == 0 '']'''
2026-03-14 09:56:26.829502 | controller |     - '    logger.go:42: 09:53:13 | watcher/5-cleanup-watcher | running command: [sh -c
2026-03-14 09:56:26.829506 | controller |       set -ex'
2026-03-14 09:56:26.829512 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.829516 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:56:26.829521 | controller |     - '        ]'
2026-03-14 09:56:26.829525 | controller |     - '    logger.go:42: 09:53:13 | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.829529 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:56:26.829535 | controller |     - '    logger.go:42: 09:53:13 | watcher/5-cleanup-watcher | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.829539 | controller |     - '    logger.go:42: 09:53:15 | watcher/5-cleanup-watcher | + ''['' 0 == 0 '']'''
2026-03-14 09:56:26.829544 | controller |     - '    logger.go:42: 09:53:16 | watcher/5-cleanup-watcher | running command: [sh -c
2026-03-14 09:56:26.829548 | controller |       set -ex'
2026-03-14 09:56:26.829552 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.829556 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:56:26.829560 | controller |     - '        ]'
2026-03-14 09:56:26.829565 | controller |     - '    logger.go:42: 09:53:16 | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default
2026-03-14 09:56:26.829569 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:56:26.829573 | controller |     - '    logger.go:42: 09:53:16 | watcher/5-cleanup-watcher | ++ grep -c ''^watcher'''
2026-03-14 09:56:26.829577 | controller |     - '    logger.go:42: 09:53:18 | watcher/5-cleanup-watcher | + ''['' 0 == 0 '']'''
2026-03-14 09:56:26.829582 | controller |     - '    logger.go:42: 09:53:18 | watcher/5-cleanup-watcher | test step completed 5-cleanup-watcher'
2026-03-14 09:56:26.829586 | controller |     - '    logger.go:42: 09:53:18 | watcher | skipping kubernetes event logging'
2026-03-14 09:56:26.829590 | controller |     - === CONT  kuttl/harness/watcher-cinder
2026-03-14 09:56:26.829608 | controller |     - '    logger.go:42: 09:53:18 | watcher-cinder | Skipping creation of user-supplied
2026-03-14 09:56:26.829614 | controller |       namespace: watcher-kuttl-default'
2026-03-14 09:56:26.829618 | controller |     - '    logger.go:42: 09:53:18 | watcher-cinder/0-cleanup-watcher | starting test step
2026-03-14 09:56:26.829622 | controller |       0-cleanup-watcher'
2026-03-14 09:56:26.829627 | controller |     - '    logger.go:42: 09:53:18 | watcher-cinder/0-cleanup-watcher | test step completed
2026-03-14 09:56:26.829631 | controller |       0-cleanup-watcher'
2026-03-14 09:56:26.829635 | controller |     - '    logger.go:42: 09:53:18 | watcher-cinder/1-deploy-watcher-no-cinder | starting
2026-03-14 09:56:26.829639 | controller |       test step 1-deploy-watcher-no-cinder'
2026-03-14 09:56:26.829650 | controller |     - '    logger.go:42: 09:53:18 | watcher-cinder/1-deploy-watcher-no-cinder | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:56:26.829654 | controller |       created'
2026-03-14 09:56:26.829659 | controller |     - '    logger.go:42: 09:53:18 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.829663 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.829667 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.829672 | controller |       service'
2026-03-14 09:56:26.829676 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.829680 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.829684 | controller |     - '        ]'
2026-03-14 09:56:26.829688 | controller |     - '    logger.go:42: 09:53:18 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.829693 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.829697 | controller |     - '    logger.go:42: 09:53:18 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.829701 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.829705 | controller |     - '    logger.go:42: 09:53:18 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:56:26.829710 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:56:26.829714 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:56:26.829718 | controller |     - '    logger.go:42: 09:53:18 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.829722 | controller |       0 == 2 '']'''
2026-03-14 09:56:26.829726 | controller |     - '    logger.go:42: 09:53:20 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.829731 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.829735 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.829739 | controller |       service'
2026-03-14 09:56:26.829743 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.829748 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.829752 | controller |     - '        ]'
2026-03-14 09:56:26.829756 | controller |     - '    logger.go:42: 09:53:20 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.829760 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.829764 | controller |     - '    logger.go:42: 09:53:20 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.829772 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.829776 | controller |     - '    logger.go:42: 09:53:20 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:56:26.829780 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:56:26.829785 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:56:26.829789 | controller |     - '    logger.go:42: 09:53:20 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.829793 | controller |       0 == 2 '']'''
2026-03-14 09:56:26.829797 | controller |     - '    logger.go:42: 09:53:21 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.829803 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.829808 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.829812 | controller |       service'
2026-03-14 09:56:26.829816 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.829820 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.829825 | controller |     - '        ]'
2026-03-14 09:56:26.829830 | controller |     - '    logger.go:42: 09:53:21 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.829835 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.829839 | controller |     - '    logger.go:42: 09:53:21 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.829843 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.829848 | controller |     - '    logger.go:42: 09:53:21 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:56:26.829852 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:56:26.829856 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:56:26.829860 | controller |     - '    logger.go:42: 09:53:21 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.829864 | controller |       0 == 2 '']'''
2026-03-14 09:56:26.829869 | controller |     - '    logger.go:42: 09:53:22 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.829873 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.829877 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.829881 | controller |       service'
2026-03-14 09:56:26.829886 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.829890 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.829894 | controller |     - '        ]'
2026-03-14 09:56:26.829898 | controller |     - '    logger.go:42: 09:53:22 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.829903 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.829907 | controller |     - '    logger.go:42: 09:53:22 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.829911 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.829921 | controller |     - '    logger.go:42: 09:53:22 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:56:26.829926 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:56:26.829930 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:56:26.829934 | controller |     - '    logger.go:42: 09:53:22 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.829939 | controller |       0 == 2 '']'''
2026-03-14 09:56:26.829943 | controller |     - '    logger.go:42: 09:53:23 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.829947 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.829951 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.829956 | controller |       service'
2026-03-14 09:56:26.829960 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.829964 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.829968 | controller |     - '        ]'
2026-03-14 09:56:26.829973 | controller |     - '    logger.go:42: 09:53:23 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.829977 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.829981 | controller |     - '    logger.go:42: 09:53:23 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.829985 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.829989 | controller |     - '    logger.go:42: 09:53:23 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:56:26.829995 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:56:26.830000 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:56:26.830004 | controller |     - '    logger.go:42: 09:53:23 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.830008 | controller |       0 == 2 '']'''
2026-03-14 09:56:26.830012 | controller |     - '    logger.go:42: 09:53:24 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.830016 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.830021 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.830025 | controller |       service'
2026-03-14 09:56:26.830029 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.830033 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.830038 | controller |     - '        ]'
2026-03-14 09:56:26.830042 | controller |     - '    logger.go:42: 09:53:24 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.830046 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.830050 | controller |     - '    logger.go:42: 09:53:24 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.830055 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.830059 | controller |     - '    logger.go:42: 09:53:24 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:56:26.830063 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:56:26.830067 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:56:26.830071 | controller |     - '    logger.go:42: 09:53:24 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.830076 | controller |       0 == 2 '']'''
2026-03-14 09:56:26.830080 | controller |     - '    logger.go:42: 09:53:25 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.830084 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.830088 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.830092 | controller |       service'
2026-03-14 09:56:26.830098 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.830103 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.830107 | controller |     - '        ]'
2026-03-14 09:56:26.830111 | controller |     - '    logger.go:42: 09:53:25 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.830115 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.830119 | controller |     - '    logger.go:42: 09:53:25 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.830124 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.830128 | controller |     - '    logger.go:42: 09:53:26 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:56:26.830132 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:56:26.830136 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:56:26.830141 | controller |     - '    logger.go:42: 09:53:26 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.830145 | controller |       0 == 2 '']'''
2026-03-14 09:56:26.830149 | controller |     - '    logger.go:42: 09:53:27 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.830153 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.830158 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.830162 | controller |       service'
2026-03-14 09:56:26.830166 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.830170 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.830175 | controller |     - '        ]'
2026-03-14 09:56:26.830179 | controller |     - '    logger.go:42: 09:53:27 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.830192 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.830202 | controller |     - '    logger.go:42: 09:53:27 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.830207 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.830211 | controller |     - '    logger.go:42: 09:53:27 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:56:26.830215 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:56:26.830219 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:56:26.830224 | controller |     - '    logger.go:42: 09:53:27 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.830228 | controller |       0 == 2 '']'''
2026-03-14 09:56:26.830232 | controller |     - '    logger.go:42: 09:53:28 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.830236 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.830241 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.830245 | controller |       service'
2026-03-14 09:56:26.830249 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.830253 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.830258 | controller |     - '        ]'
2026-03-14 09:56:26.830262 | controller |     - '    logger.go:42: 09:53:28 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.830266 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.830270 | controller |     - '    logger.go:42: 09:53:28 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.830275 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.830281 | controller |     - '    logger.go:42: 09:53:28 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:56:26.830285 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:56:26.830289 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:56:26.830294 | controller |     - '    logger.go:42: 09:53:28 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.830298 | controller |       0 == 2 '']'''
2026-03-14 09:56:26.830304 | controller |     - '    logger.go:42: 09:53:29 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.830308 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.830312 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.830316 | controller |       service'
2026-03-14 09:56:26.830321 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.830325 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.830329 | controller |     - '        ]'
2026-03-14 09:56:26.830333 | controller |     - '    logger.go:42: 09:53:29 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.830338 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.830342 | controller |     - '    logger.go:42: 09:53:29 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.830346 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.830350 | controller |     - '    logger.go:42: 09:53:29 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:56:26.830355 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:56:26.830359 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:56:26.830363 | controller |     - '    logger.go:42: 09:53:29 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.830367 | controller |       0 == 2 '']'''
2026-03-14 09:56:26.830372 | controller |     - '    logger.go:42: 09:53:30 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.830376 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.830383 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.830388 | controller |       service'
2026-03-14 09:56:26.830392 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.830396 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.830400 | controller |     - '        ]'
2026-03-14 09:56:26.830405 | controller |     - '    logger.go:42: 09:53:30 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.830409 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.830413 | controller |     - '    logger.go:42: 09:53:30 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.830417 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.830421 | controller |     - '    logger.go:42: 09:53:30 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:56:26.830426 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:56:26.830430 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:56:26.830434 | controller |     - '    logger.go:42: 09:53:30 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.830438 | controller |       0 == 2 '']'''
2026-03-14 09:56:26.830442 | controller |     - '    logger.go:42: 09:53:31 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.830447 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.830451 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.830455 | controller |       service'
2026-03-14 09:56:26.830459 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.830464 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.830468 | controller |     - '        ]'
2026-03-14 09:56:26.830478 | controller |     - '    logger.go:42: 09:53:31 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.830483 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.830487 | controller |     - '    logger.go:42: 09:53:31 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.830491 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.830495 | controller |     - '    logger.go:42: 09:53:31 | watcher-cinder/1-deploy-watcher-no-cinder | Error
2026-03-14 09:56:26.830500 | controller |       from server (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0"
2026-03-14 09:56:26.830504 | controller |       is waiting to start: ContainerCreating'
2026-03-14 09:56:26.830508 | controller |     - '    logger.go:42: 09:53:31 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.830512 | controller |       0 == 2 '']'''
2026-03-14 09:56:26.830517 | controller |     - '    logger.go:42: 09:53:32 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.830521 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.830525 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.830529 | controller |       service'
2026-03-14 09:56:26.830534 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.830538 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.830542 | controller |     - '        ]'
2026-03-14 09:56:26.830546 | controller |     - '    logger.go:42: 09:53:32 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.830551 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.830555 | controller |     - '    logger.go:42: 09:53:32 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.830559 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.830563 | controller |     - '    logger.go:42: 09:53:33 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.830569 | controller |       0 == 2 '']'''
2026-03-14 09:56:26.830573 | controller |     - '    logger.go:42: 09:53:34 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.830578 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.830582 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.830586 | controller |       service'
2026-03-14 09:56:26.830590 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.830606 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.830612 | controller |     - '        ]'
2026-03-14 09:56:26.830616 | controller |     - '    logger.go:42: 09:53:34 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.830620 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.830624 | controller |     - '    logger.go:42: 09:53:34 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.830630 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.830635 | controller |     - '    logger.go:42: 09:53:34 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.830639 | controller |       0 == 2 '']'''
2026-03-14 09:56:26.830643 | controller |     - '    logger.go:42: 09:53:35 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.830647 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.830652 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.830656 | controller |       service'
2026-03-14 09:56:26.830660 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.830664 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.830669 | controller |     - '        ]'
2026-03-14 09:56:26.830673 | controller |     - '    logger.go:42: 09:53:35 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.830679 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.830685 | controller |     - '    logger.go:42: 09:53:35 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.830691 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.830699 | controller |     - '    logger.go:42: 09:53:35 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.830705 | controller |       0 == 2 '']'''
2026-03-14 09:56:26.830710 | controller |     - '    logger.go:42: 09:53:36 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.830714 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.830719 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.830723 | controller |       service'
2026-03-14 09:56:26.830727 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.830731 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.830735 | controller |     - '        ]'
2026-03-14 09:56:26.830740 | controller |     - '    logger.go:42: 09:53:36 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.830744 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.830748 | controller |     - '    logger.go:42: 09:53:36 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.830752 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.830757 | controller |     - '    logger.go:42: 09:53:36 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.830761 | controller |       2 == 2 '']'''
2026-03-14 09:56:26.830773 | controller |     - '    logger.go:42: 09:53:37 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.830777 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.830782 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.830786 | controller |       service'
2026-03-14 09:56:26.830792 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.830797 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.830801 | controller |     - '        ]'
2026-03-14 09:56:26.830805 | controller |     - '    logger.go:42: 09:53:37 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.830809 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.830813 | controller |     - '    logger.go:42: 09:53:37 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.830818 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.830822 | controller |     - '    logger.go:42: 09:53:37 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.830826 | controller |       2 == 2 '']'''
2026-03-14 09:56:26.830830 | controller |     - '    logger.go:42: 09:53:38 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.830835 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.830839 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.830843 | controller |       service'
2026-03-14 09:56:26.830847 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.830852 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.830856 | controller |     - '        ]'
2026-03-14 09:56:26.830860 | controller |     - '    logger.go:42: 09:53:38 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.830865 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.830869 | controller |     - '    logger.go:42: 09:53:38 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.830873 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.830878 | controller |     - '    logger.go:42: 09:53:39 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.830882 | controller |       2 == 2 '']'''
2026-03-14 09:56:26.830886 | controller |     - '    logger.go:42: 09:53:40 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.830890 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.830895 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.830899 | controller |       service'
2026-03-14 09:56:26.830903 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.830908 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.830912 | controller |     - '        ]'
2026-03-14 09:56:26.830916 | controller |     - '    logger.go:42: 09:53:40 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.830921 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.830925 | controller |     - '    logger.go:42: 09:53:40 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.830929 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.830934 | controller |     - '    logger.go:42: 09:53:40 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.830938 | controller |       2 == 2 '']'''
2026-03-14 09:56:26.830942 | controller |     - '    logger.go:42: 09:53:41 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.830946 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.830951 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.830955 | controller |       service'
2026-03-14 09:56:26.830959 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.830964 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.830968 | controller |     - '        ]'
2026-03-14 09:56:26.830972 | controller |     - '    logger.go:42: 09:53:41 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.830978 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.830983 | controller |     - '    logger.go:42: 09:53:41 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.830987 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.830991 | controller |     - '    logger.go:42: 09:53:41 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.830995 | controller |       2 == 2 '']'''
2026-03-14 09:56:26.831003 | controller |     - '    logger.go:42: 09:53:42 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:56:26.831008 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.831012 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.831017 | controller |       service'
2026-03-14 09:56:26.831021 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:56:26.831025 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.831029 | controller |     - '        ]'
2026-03-14 09:56:26.831033 | controller |     - '    logger.go:42: 09:53:42 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:56:26.831038 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.831042 | controller |     - '    logger.go:42: 09:53:42 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:56:26.831062 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.831069 | controller |     - '    logger.go:42: 09:53:42 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:56:26.831074 | controller |       2 == 2 '']'''
2026-03-14 09:56:26.831078 | controller |     - '    logger.go:42: 09:53:42 | watcher-cinder/1-deploy-watcher-no-cinder | test step
2026-03-14 09:56:26.831082 | controller |       completed 1-deploy-watcher-no-cinder'
2026-03-14 09:56:26.831087 | controller |     - '    logger.go:42: 09:53:42 | watcher-cinder/2-deploy-cinder | starting test step
2026-03-14 09:56:26.831091 | controller |       2-deploy-cinder'
2026-03-14 09:56:26.831095 | controller |     - '    logger.go:42: 09:53:42 | watcher-cinder/2-deploy-cinder | OpenStackControlPlane:watcher-kuttl-default/openstack
2026-03-14 09:56:26.831100 | controller |       updated'
2026-03-14 09:56:26.831106 | controller |     - '    logger.go:42: 09:53:42 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.831112 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.831121 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.831126 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.831132 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.831138 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.831143 | controller |     - '        ]'
2026-03-14 09:56:26.831149 | controller |     - '    logger.go:42: 09:53:42 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.831155 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.831160 | controller |     - '    logger.go:42: 09:53:42 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.831165 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.831169 | controller |     - '    logger.go:42: 09:53:43 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.831173 | controller |     - '    logger.go:42: 09:53:44 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.831178 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.831182 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.831186 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.831191 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.831195 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.831199 | controller |     - '        ]'
2026-03-14 09:56:26.831206 | controller |     - '    logger.go:42: 09:53:44 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.831210 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.831214 | controller |     - '    logger.go:42: 09:53:44 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.831219 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.831223 | controller |     - '    logger.go:42: 09:53:44 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.831227 | controller |     - '    logger.go:42: 09:53:45 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.831231 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.831236 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.831240 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.831244 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.831248 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.831252 | controller |     - '        ]'
2026-03-14 09:56:26.831257 | controller |     - '    logger.go:42: 09:53:45 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.831261 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.831265 | controller |     - '    logger.go:42: 09:53:45 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.831269 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.831274 | controller |     - '    logger.go:42: 09:53:45 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.831278 | controller |     - '    logger.go:42: 09:53:46 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.831282 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.831286 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.831290 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.831295 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.831299 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.831303 | controller |     - '        ]'
2026-03-14 09:56:26.831307 | controller |     - '    logger.go:42: 09:53:46 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.831312 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.831316 | controller |     - '    logger.go:42: 09:53:46 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.831320 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.831324 | controller |     - '    logger.go:42: 09:53:46 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.831329 | controller |     - '    logger.go:42: 09:53:47 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.831333 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.831344 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.831349 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.831353 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.831358 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.831362 | controller |     - '        ]'
2026-03-14 09:56:26.831366 | controller |     - '    logger.go:42: 09:53:47 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.831371 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.831378 | controller |     - '    logger.go:42: 09:53:47 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.831384 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.831388 | controller |     - '    logger.go:42: 09:53:48 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.831394 | controller |     - '    logger.go:42: 09:53:49 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.831398 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.831402 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.831407 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.831411 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.831415 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.831419 | controller |     - '        ]'
2026-03-14 09:56:26.831424 | controller |     - '    logger.go:42: 09:53:49 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.831428 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.831432 | controller |     - '    logger.go:42: 09:53:49 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.831437 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.831441 | controller |     - '    logger.go:42: 09:53:49 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.831445 | controller |     - '    logger.go:42: 09:53:50 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.831449 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.831454 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.831458 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.831462 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.831466 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.831471 | controller |     - '        ]'
2026-03-14 09:56:26.831475 | controller |     - '    logger.go:42: 09:53:50 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.831479 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.831483 | controller |     - '    logger.go:42: 09:53:50 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.831488 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.831492 | controller |     - '    logger.go:42: 09:53:50 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.831496 | controller |     - '    logger.go:42: 09:53:51 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.831500 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.831505 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.831509 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.831513 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.831517 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.831522 | controller |     - '        ]'
2026-03-14 09:56:26.831526 | controller |     - '    logger.go:42: 09:53:51 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.831530 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.831534 | controller |     - '    logger.go:42: 09:53:51 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.831539 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.831543 | controller |     - '    logger.go:42: 09:53:51 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.831547 | controller |     - '    logger.go:42: 09:53:52 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.831551 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.831556 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.831560 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.831564 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.831570 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.831574 | controller |     - '        ]'
2026-03-14 09:56:26.831579 | controller |     - '    logger.go:42: 09:53:52 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.831583 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.831587 | controller |     - '    logger.go:42: 09:53:52 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.831591 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.831609 | controller |     - '    logger.go:42: 09:53:52 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.831620 | controller |     - '    logger.go:42: 09:53:53 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.831625 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.831629 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.831633 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.831638 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.831642 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.831646 | controller |     - '        ]'
2026-03-14 09:56:26.831650 | controller |     - '    logger.go:42: 09:53:53 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.831655 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.831659 | controller |     - '    logger.go:42: 09:53:53 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.831663 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.831667 | controller |     - '    logger.go:42: 09:53:54 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.831671 | controller |     - '    logger.go:42: 09:53:55 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.831676 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.831680 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.831684 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.831688 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.831692 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.831697 | controller |     - '        ]'
2026-03-14 09:56:26.831701 | controller |     - '    logger.go:42: 09:53:55 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.831705 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.831709 | controller |     - '    logger.go:42: 09:53:55 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.831714 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.831720 | controller |     - '    logger.go:42: 09:53:55 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.831724 | controller |     - '    logger.go:42: 09:53:56 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.831728 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.831733 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.831737 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.831741 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.831745 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.831750 | controller |     - '        ]'
2026-03-14 09:56:26.831754 | controller |     - '    logger.go:42: 09:53:56 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.831758 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.831762 | controller |     - '    logger.go:42: 09:53:56 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.831768 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.831773 | controller |     - '    logger.go:42: 09:53:56 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.831777 | controller |     - '    logger.go:42: 09:53:57 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.831781 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.831785 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.831790 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.831796 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.831802 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.831807 | controller |     - '        ]'
2026-03-14 09:56:26.831813 | controller |     - '    logger.go:42: 09:53:57 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.831818 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.831823 | controller |     - '    logger.go:42: 09:53:57 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.831829 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.831835 | controller |     - '    logger.go:42: 09:53:57 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.831840 | controller |     - '    logger.go:42: 09:53:58 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.831846 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.831851 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.831857 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.831862 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.831868 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.831873 | controller |     - '        ]'
2026-03-14 09:56:26.831879 | controller |     - '    logger.go:42: 09:53:58 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.831885 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.831890 | controller |     - '    logger.go:42: 09:53:58 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.831896 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.831910 | controller |     - '    logger.go:42: 09:53:59 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.831917 | controller |     - '    logger.go:42: 09:54:00 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.831923 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.831928 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.831934 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.831940 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.831945 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.831949 | controller |     - '        ]'
2026-03-14 09:56:26.831954 | controller |     - '    logger.go:42: 09:54:00 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.831958 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.831962 | controller |     - '    logger.go:42: 09:54:00 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.831967 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.831971 | controller |     - '    logger.go:42: 09:54:00 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.831975 | controller |     - '    logger.go:42: 09:54:01 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.831979 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.831984 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.831990 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.831995 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.831999 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.832003 | controller |     - '        ]'
2026-03-14 09:56:26.832007 | controller |     - '    logger.go:42: 09:54:01 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.832012 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.832016 | controller |     - '    logger.go:42: 09:54:01 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.832020 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.832025 | controller |     - '    logger.go:42: 09:54:01 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.832030 | controller |     - '    logger.go:42: 09:54:02 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.832036 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.832041 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.832047 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.832052 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.832056 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.832060 | controller |     - '        ]'
2026-03-14 09:56:26.832065 | controller |     - '    logger.go:42: 09:54:02 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.832069 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.832076 | controller |     - '    logger.go:42: 09:54:02 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.832081 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.832085 | controller |     - '    logger.go:42: 09:54:02 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.832091 | controller |     - '    logger.go:42: 09:54:03 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.832095 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.832099 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.832104 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.832108 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.832112 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.832116 | controller |     - '        ]'
2026-03-14 09:56:26.832121 | controller |     - '    logger.go:42: 09:54:03 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.832125 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.832129 | controller |     - '    logger.go:42: 09:54:03 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.832133 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.832138 | controller |     - '    logger.go:42: 09:54:04 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.832142 | controller |     - '    logger.go:42: 09:54:05 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.832146 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.832150 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.832154 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.832159 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.832163 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.832167 | controller |     - '        ]'
2026-03-14 09:56:26.832171 | controller |     - '    logger.go:42: 09:54:05 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.832175 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.832189 | controller |     - '    logger.go:42: 09:54:05 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.832193 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.832198 | controller |     - '    logger.go:42: 09:54:05 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.832202 | controller |     - '    logger.go:42: 09:54:06 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.832206 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.832211 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.832215 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.832219 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.832223 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.832227 | controller |     - '        ]'
2026-03-14 09:56:26.832232 | controller |     - '    logger.go:42: 09:54:06 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.832236 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.832240 | controller |     - '    logger.go:42: 09:54:06 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.832244 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.832249 | controller |     - '    logger.go:42: 09:54:06 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.832253 | controller |     - '    logger.go:42: 09:54:07 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.832257 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.832261 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.832266 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.832270 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.832274 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.832279 | controller |     - '        ]'
2026-03-14 09:56:26.832283 | controller |     - '    logger.go:42: 09:54:07 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.832287 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.832291 | controller |     - '    logger.go:42: 09:54:07 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.832296 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.832300 | controller |     - '    logger.go:42: 09:54:07 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.832304 | controller |     - '    logger.go:42: 09:54:08 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.832309 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.832313 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.832317 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.832322 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.832326 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.832330 | controller |     - '        ]'
2026-03-14 09:56:26.832335 | controller |     - '    logger.go:42: 09:54:08 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.832339 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.832343 | controller |     - '    logger.go:42: 09:54:08 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.832348 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.832352 | controller |     - '    logger.go:42: 09:54:09 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.832356 | controller |     - '    logger.go:42: 09:54:10 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.832361 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.832367 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.832371 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.832375 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.832380 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.832384 | controller |     - '        ]'
2026-03-14 09:56:26.832388 | controller |     - '    logger.go:42: 09:54:10 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.832393 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.832397 | controller |     - '    logger.go:42: 09:54:10 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.832401 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.832405 | controller |     - '    logger.go:42: 09:54:10 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.832410 | controller |     - '    logger.go:42: 09:54:11 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.832414 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.832420 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.832425 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.832429 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.832433 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.832438 | controller |     - '        ]'
2026-03-14 09:56:26.832448 | controller |     - '    logger.go:42: 09:54:11 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.832452 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.832457 | controller |     - '    logger.go:42: 09:54:11 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.832461 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.832465 | controller |     - '    logger.go:42: 09:54:11 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.832470 | controller |     - '    logger.go:42: 09:54:12 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.832474 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.832478 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.832483 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.832487 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.832491 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.832496 | controller |     - '        ]'
2026-03-14 09:56:26.832500 | controller |     - '    logger.go:42: 09:54:12 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.832504 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.832509 | controller |     - '    logger.go:42: 09:54:12 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.832513 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.832526 | controller |     - '    logger.go:42: 09:54:12 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.832531 | controller |     - '    logger.go:42: 09:54:13 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.832535 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.832540 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.832544 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.832549 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.832553 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.832558 | controller |     - '        ]'
2026-03-14 09:56:26.832564 | controller |     - '    logger.go:42: 09:54:13 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.832570 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.832574 | controller |     - '    logger.go:42: 09:54:13 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.832579 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.832583 | controller |     - '    logger.go:42: 09:54:14 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.832588 | controller |     - '    logger.go:42: 09:54:15 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.832592 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.832611 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.832615 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.832620 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.832624 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.832628 | controller |     - '        ]'
2026-03-14 09:56:26.832632 | controller |     - '    logger.go:42: 09:54:15 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.832637 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.832641 | controller |     - '    logger.go:42: 09:54:15 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.832646 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.832650 | controller |     - '    logger.go:42: 09:54:15 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.832654 | controller |     - '    logger.go:42: 09:54:16 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.832658 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.832663 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.832667 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.832671 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.832676 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.832680 | controller |     - '        ]'
2026-03-14 09:56:26.832684 | controller |     - '    logger.go:42: 09:54:16 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.832689 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.832693 | controller |     - '    logger.go:42: 09:54:16 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.832697 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.832702 | controller |     - '    logger.go:42: 09:54:16 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.832708 | controller |     - '    logger.go:42: 09:54:17 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.832712 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.832717 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.832721 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.832725 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.832737 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.832741 | controller |     - '        ]'
2026-03-14 09:56:26.832746 | controller |     - '    logger.go:42: 09:54:17 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.832750 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.832756 | controller |     - '    logger.go:42: 09:54:17 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.832761 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.832765 | controller |     - '    logger.go:42: 09:54:17 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.832779 | controller |     - '    logger.go:42: 09:54:18 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.832783 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.832788 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.832792 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.832798 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.832802 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.832807 | controller |     - '        ]'
2026-03-14 09:56:26.832811 | controller |     - '    logger.go:42: 09:54:18 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.832815 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.832820 | controller |     - '    logger.go:42: 09:54:18 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.832824 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.832828 | controller |     - '    logger.go:42: 09:54:18 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.832833 | controller |     - '    logger.go:42: 09:54:19 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.832837 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.832841 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.832846 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.832850 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.832854 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.832859 | controller |     - '        ]'
2026-03-14 09:56:26.832863 | controller |     - '    logger.go:42: 09:54:19 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.832867 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.832872 | controller |     - '    logger.go:42: 09:54:19 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.832876 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.832880 | controller |     - '    logger.go:42: 09:54:20 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.832885 | controller |     - '    logger.go:42: 09:54:21 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.832889 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.832893 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.832898 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.832902 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.832906 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.832910 | controller |     - '        ]'
2026-03-14 09:56:26.832915 | controller |     - '    logger.go:42: 09:54:21 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.832919 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.832923 | controller |     - '    logger.go:42: 09:54:21 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.832928 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.832932 | controller |     - '    logger.go:42: 09:54:21 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.832936 | controller |     - '    logger.go:42: 09:54:22 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.832941 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.832945 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.832949 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.832954 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.832960 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.832964 | controller |     - '        ]'
2026-03-14 09:56:26.832968 | controller |     - '    logger.go:42: 09:54:22 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.832973 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.832977 | controller |     - '    logger.go:42: 09:54:22 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.832981 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.832985 | controller |     - '    logger.go:42: 09:54:22 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.832990 | controller |     - '    logger.go:42: 09:54:23 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.832994 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.832998 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.833009 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.833014 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.833018 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.833022 | controller |     - '        ]'
2026-03-14 09:56:26.833027 | controller |     - '    logger.go:42: 09:54:23 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.833031 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.833036 | controller |     - '    logger.go:42: 09:54:23 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.833040 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.833044 | controller |     - '    logger.go:42: 09:54:23 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.833049 | controller |     - '    logger.go:42: 09:54:24 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.833053 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.833057 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.833062 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.833066 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.833070 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.833074 | controller |     - '        ]'
2026-03-14 09:56:26.833079 | controller |     - '    logger.go:42: 09:54:24 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.833083 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.833087 | controller |     - '    logger.go:42: 09:54:24 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.833091 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.833096 | controller |     - '    logger.go:42: 09:54:25 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.833100 | controller |     - '    logger.go:42: 09:54:26 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.833104 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.833108 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.833113 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.833117 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.833121 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.833125 | controller |     - '        ]'
2026-03-14 09:56:26.833130 | controller |     - '    logger.go:42: 09:54:26 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.833134 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.833140 | controller |     - '    logger.go:42: 09:54:26 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.833145 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.833151 | controller |     - '    logger.go:42: 09:54:26 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.833155 | controller |     - '    logger.go:42: 09:54:27 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.833159 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.833163 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.833168 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.833172 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.833176 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.833180 | controller |     - '        ]'
2026-03-14 09:56:26.833185 | controller |     - '    logger.go:42: 09:54:27 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.833189 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.833193 | controller |     - '    logger.go:42: 09:54:27 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.833197 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.833202 | controller |     - '    logger.go:42: 09:54:27 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.833206 | controller |     - '    logger.go:42: 09:54:28 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.833210 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.833214 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.833218 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.833223 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.833227 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.833231 | controller |     - '        ]'
2026-03-14 09:56:26.833238 | controller |     - '    logger.go:42: 09:54:28 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.833242 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.833246 | controller |     - '    logger.go:42: 09:54:28 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.833250 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.833255 | controller |     - '    logger.go:42: 09:54:28 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.833259 | controller |     - '    logger.go:42: 09:54:29 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.833269 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.833274 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.833278 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.833282 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.833287 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.833291 | controller |     - '        ]'
2026-03-14 09:56:26.833295 | controller |     - '    logger.go:42: 09:54:29 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.833300 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.833304 | controller |     - '    logger.go:42: 09:54:29 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.833308 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.833313 | controller |     - '    logger.go:42: 09:54:29 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.833317 | controller |     - '    logger.go:42: 09:54:30 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.833321 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.833325 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.833330 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.833337 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.833341 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.833346 | controller |     - '        ]'
2026-03-14 09:56:26.833350 | controller |     - '    logger.go:42: 09:54:30 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.833354 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.833359 | controller |     - '    logger.go:42: 09:54:30 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.833363 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.833367 | controller |     - '    logger.go:42: 09:54:31 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.833371 | controller |     - '    logger.go:42: 09:54:32 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.833376 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.833380 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.833384 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.833388 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.833393 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.833397 | controller |     - '        ]'
2026-03-14 09:56:26.833401 | controller |     - '    logger.go:42: 09:54:32 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.833408 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.833412 | controller |     - '    logger.go:42: 09:54:32 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.833416 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.833421 | controller |     - '    logger.go:42: 09:54:32 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.833425 | controller |     - '    logger.go:42: 09:54:33 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.833429 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.833433 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.833438 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.833442 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.833446 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.833450 | controller |     - '        ]'
2026-03-14 09:56:26.833455 | controller |     - '    logger.go:42: 09:54:33 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.833459 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.833463 | controller |     - '    logger.go:42: 09:54:33 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.833468 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.833473 | controller |     - '    logger.go:42: 09:54:33 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.833478 | controller |     - '    logger.go:42: 09:54:34 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.833482 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.833486 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.833491 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.833495 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.833499 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.833503 | controller |     - '        ]'
2026-03-14 09:56:26.833508 | controller |     - '    logger.go:42: 09:54:34 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.833512 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.833518 | controller |     - '    logger.go:42: 09:54:34 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.833523 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.833529 | controller |     - '    logger.go:42: 09:54:34 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.833533 | controller |     - '    logger.go:42: 09:54:35 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.833538 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.833542 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.833546 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.833550 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.833555 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.833559 | controller |     - '        ]'
2026-03-14 09:56:26.833563 | controller |     - '    logger.go:42: 09:54:35 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.833567 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.833571 | controller |     - '    logger.go:42: 09:54:35 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.833576 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.833580 | controller |     - '    logger.go:42: 09:54:36 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.833584 | controller |     - '    logger.go:42: 09:54:37 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.833588 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.833604 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.833610 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.833630 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.833634 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.833638 | controller |     - '        ]'
2026-03-14 09:56:26.833643 | controller |     - '    logger.go:42: 09:54:37 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.833647 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.833651 | controller |     - '    logger.go:42: 09:54:37 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.833655 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.833660 | controller |     - '    logger.go:42: 09:54:37 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.833664 | controller |     - '    logger.go:42: 09:54:38 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.833668 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.833672 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.833677 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.833681 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.833685 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.833689 | controller |     - '        ]'
2026-03-14 09:56:26.833694 | controller |     - '    logger.go:42: 09:54:38 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.833698 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.833702 | controller |     - '    logger.go:42: 09:54:38 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.833706 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.833711 | controller |     - '    logger.go:42: 09:54:38 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.833715 | controller |     - '    logger.go:42: 09:54:39 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.833719 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.833727 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.833731 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.833736 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.833740 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.833744 | controller |     - '        ]'
2026-03-14 09:56:26.833748 | controller |     - '    logger.go:42: 09:54:39 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.833753 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.833757 | controller |     - '    logger.go:42: 09:54:39 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.833761 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.833765 | controller |     - '    logger.go:42: 09:54:39 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.833770 | controller |     - '    logger.go:42: 09:54:40 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.833774 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.833778 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.833782 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.833787 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.833791 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.833795 | controller |     - '        ]'
2026-03-14 09:56:26.833799 | controller |     - '    logger.go:42: 09:54:40 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.833803 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.833811 | controller |     - '    logger.go:42: 09:54:40 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.833816 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.833820 | controller |     - '    logger.go:42: 09:54:41 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:56:26.833826 | controller |     - '    logger.go:42: 09:54:42 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.833831 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.833835 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.833839 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.833844 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.833848 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.833852 | controller |     - '        ]'
2026-03-14 09:56:26.833856 | controller |     - '    logger.go:42: 09:54:42 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.833860 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.833865 | controller |     - '    logger.go:42: 09:54:42 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.833869 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.833873 | controller |     - '    logger.go:42: 09:54:42 | watcher-cinder/2-deploy-cinder | Error from server
2026-03-14 09:56:26.833877 | controller |       (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0"
2026-03-14 09:56:26.833882 | controller |       is waiting to start: ContainerCreating'
2026-03-14 09:56:26.833886 | controller |     - '    logger.go:42: 09:54:42 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:56:26.833890 | controller |     - '    logger.go:42: 09:54:43 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.833894 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.833899 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.833903 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.833909 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.833913 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.833917 | controller |     - '        ]'
2026-03-14 09:56:26.833922 | controller |     - '    logger.go:42: 09:54:43 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.833926 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.833932 | controller |     - '    logger.go:42: 09:54:43 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.833936 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.833941 | controller |     - '    logger.go:42: 09:54:43 | watcher-cinder/2-deploy-cinder | Error from server
2026-03-14 09:56:26.833945 | controller |       (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0"
2026-03-14 09:56:26.833949 | controller |       is waiting to start: ContainerCreating'
2026-03-14 09:56:26.833954 | controller |     - '    logger.go:42: 09:54:43 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:56:26.833958 | controller |     - '    logger.go:42: 09:54:44 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.833962 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.833966 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.833971 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.833975 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.833979 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.833983 | controller |     - '        ]'
2026-03-14 09:56:26.833988 | controller |     - '    logger.go:42: 09:54:44 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.833992 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.833996 | controller |     - '    logger.go:42: 09:54:44 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.834000 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.834006 | controller |     - '    logger.go:42: 09:54:44 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:56:26.834010 | controller |     - '    logger.go:42: 09:54:45 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.834015 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.834019 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.834023 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.834027 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.834032 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.834036 | controller |     - '        ]'
2026-03-14 09:56:26.834040 | controller |     - '    logger.go:42: 09:54:45 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.834044 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.834049 | controller |     - '    logger.go:42: 09:54:45 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.834053 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.834057 | controller |     - '    logger.go:42: 09:54:45 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:56:26.834061 | controller |     - '    logger.go:42: 09:54:46 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.834068 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.834073 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.834077 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.834081 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.834085 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.834091 | controller |     - '        ]'
2026-03-14 09:56:26.834096 | controller |     - '    logger.go:42: 09:54:46 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.834102 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.834106 | controller |     - '    logger.go:42: 09:54:46 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.834110 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.834115 | controller |     - '    logger.go:42: 09:54:47 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:56:26.834119 | controller |     - '    logger.go:42: 09:54:48 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.834123 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.834127 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.834131 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.834136 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.834140 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.834144 | controller |     - '        ]'
2026-03-14 09:56:26.834148 | controller |     - '    logger.go:42: 09:54:48 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.834152 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.834157 | controller |     - '    logger.go:42: 09:54:48 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.834161 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.834167 | controller |     - '    logger.go:42: 09:54:48 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:56:26.834171 | controller |     - '    logger.go:42: 09:54:49 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.834175 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.834180 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.834184 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.834188 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.834192 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.834196 | controller |     - '        ]'
2026-03-14 09:56:26.834201 | controller |     - '    logger.go:42: 09:54:49 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.834205 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.834209 | controller |     - '    logger.go:42: 09:54:49 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.834213 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.834218 | controller |     - '    logger.go:42: 09:54:49 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:56:26.834222 | controller |     - '    logger.go:42: 09:54:50 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.834226 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.834230 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.834234 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.834239 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.834243 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.834247 | controller |     - '        ]'
2026-03-14 09:56:26.834251 | controller |     - '    logger.go:42: 09:54:50 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.834256 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.834260 | controller |     - '    logger.go:42: 09:54:50 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.834264 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.834270 | controller |     - '    logger.go:42: 09:54:50 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:56:26.834274 | controller |     - '    logger.go:42: 09:54:51 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.834278 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.834283 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.834287 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.834291 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.834295 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.834300 | controller |     - '        ]'
2026-03-14 09:56:26.834304 | controller |     - '    logger.go:42: 09:54:51 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.834308 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.834312 | controller |     - '    logger.go:42: 09:54:51 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.834317 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.834323 | controller |     - '    logger.go:42: 09:54:52 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:56:26.834328 | controller |     - '    logger.go:42: 09:54:53 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.834332 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.834336 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.834340 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.834345 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.834349 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.834353 | controller |     - '        ]'
2026-03-14 09:56:26.834357 | controller |     - '    logger.go:42: 09:54:53 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.834362 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.834366 | controller |     - '    logger.go:42: 09:54:53 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.834370 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.834374 | controller |     - '    logger.go:42: 09:54:53 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:56:26.834379 | controller |     - '    logger.go:42: 09:54:54 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:56:26.834383 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.834387 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:56:26.834391 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:56:26.834396 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.834400 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:56:26.834404 | controller |     - '        ]'
2026-03-14 09:56:26.834408 | controller |     - '    logger.go:42: 09:54:54 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.834413 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.834417 | controller |     - '    logger.go:42: 09:54:54 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.834421 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.834425 | controller |     - '    logger.go:42: 09:54:54 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:56:26.834430 | controller |     - '    logger.go:42: 09:54:54 | watcher-cinder/2-deploy-cinder | test step completed
2026-03-14 09:56:26.834434 | controller |       2-deploy-cinder'
2026-03-14 09:56:26.834438 | controller |     - '    logger.go:42: 09:54:54 | watcher-cinder/3-remove-cinder | starting test step
2026-03-14 09:56:26.834442 | controller |       3-remove-cinder'
2026-03-14 09:56:26.834447 | controller |     - '    logger.go:42: 09:54:54 | watcher-cinder/3-remove-cinder | OpenStackControlPlane:watcher-kuttl-default/openstack
2026-03-14 09:56:26.834453 | controller |       updated'
2026-03-14 09:56:26.834457 | controller |     - '    logger.go:42: 09:54:54 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.834461 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.834466 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.834470 | controller |       service'
2026-03-14 09:56:26.834474 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.834478 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.834483 | controller |     - '        ]'
2026-03-14 09:56:26.834487 | controller |     - '    logger.go:42: 09:54:54 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.834491 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.834497 | controller |     - '    logger.go:42: 09:54:54 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.834501 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.834506 | controller |     - '    logger.go:42: 09:54:54 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:56:26.834510 | controller |     - '    logger.go:42: 09:54:55 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.834514 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.834518 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.834523 | controller |       service'
2026-03-14 09:56:26.834527 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.834531 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.834535 | controller |     - '        ]'
2026-03-14 09:56:26.834540 | controller |     - '    logger.go:42: 09:54:55 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.834544 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.834548 | controller |     - '    logger.go:42: 09:54:55 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.834552 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.834557 | controller |     - '    logger.go:42: 09:54:55 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:56:26.834561 | controller |     - '    logger.go:42: 09:54:57 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.834565 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.834569 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.834573 | controller |       service'
2026-03-14 09:56:26.834578 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.834582 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.834588 | controller |     - '        ]'
2026-03-14 09:56:26.834604 | controller |     - '    logger.go:42: 09:54:57 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.834611 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.834615 | controller |     - '    logger.go:42: 09:54:57 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.834619 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.834623 | controller |     - '    logger.go:42: 09:54:57 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:56:26.834628 | controller |     - '    logger.go:42: 09:54:58 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.834632 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.834636 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.834641 | controller |       service'
2026-03-14 09:56:26.834645 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.834649 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.834655 | controller |     - '        ]'
2026-03-14 09:56:26.834659 | controller |     - '    logger.go:42: 09:54:58 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.834664 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.834668 | controller |     - '    logger.go:42: 09:54:58 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.834672 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.834676 | controller |     - '    logger.go:42: 09:54:58 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:56:26.834681 | controller |     - '    logger.go:42: 09:54:59 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.834685 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.834689 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.834693 | controller |       service'
2026-03-14 09:56:26.834698 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.834702 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.834706 | controller |     - '        ]'
2026-03-14 09:56:26.834710 | controller |     - '    logger.go:42: 09:54:59 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.834715 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.834719 | controller |     - '    logger.go:42: 09:54:59 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.834723 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.834727 | controller |     - '    logger.go:42: 09:54:59 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:56:26.834731 | controller |     - '    logger.go:42: 09:55:00 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.834736 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.834740 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.834744 | controller |       service'
2026-03-14 09:56:26.834748 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.834753 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.834757 | controller |     - '        ]'
2026-03-14 09:56:26.834761 | controller |     - '    logger.go:42: 09:55:00 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.834765 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.834770 | controller |     - '    logger.go:42: 09:55:00 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.834774 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.834778 | controller |     - '    logger.go:42: 09:55:00 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:56:26.834782 | controller |     - '    logger.go:42: 09:55:01 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.834787 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.834791 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.834795 | controller |       service'
2026-03-14 09:56:26.834799 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.834803 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.834808 | controller |     - '        ]'
2026-03-14 09:56:26.834812 | controller |     - '    logger.go:42: 09:55:01 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.834816 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.834820 | controller |     - '    logger.go:42: 09:55:01 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.834825 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.834831 | controller |     - '    logger.go:42: 09:55:02 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:56:26.834837 | controller |     - '    logger.go:42: 09:55:03 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.834841 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.834845 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.834849 | controller |       service'
2026-03-14 09:56:26.834854 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.834858 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.834862 | controller |     - '        ]'
2026-03-14 09:56:26.834869 | controller |     - '    logger.go:42: 09:55:03 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.834873 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.834878 | controller |     - '    logger.go:42: 09:55:03 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.834882 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.834886 | controller |     - '    logger.go:42: 09:55:03 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:56:26.834890 | controller |     - '    logger.go:42: 09:55:04 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.834895 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.834899 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.834903 | controller |       service'
2026-03-14 09:56:26.834907 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.834912 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.834916 | controller |     - '        ]'
2026-03-14 09:56:26.834920 | controller |     - '    logger.go:42: 09:55:04 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.834925 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.834931 | controller |     - '    logger.go:42: 09:55:04 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.834935 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.834939 | controller |     - '    logger.go:42: 09:55:04 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:56:26.834944 | controller |     - '    logger.go:42: 09:55:05 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.834948 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.834952 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.834956 | controller |       service'
2026-03-14 09:56:26.834960 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.834965 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.834969 | controller |     - '        ]'
2026-03-14 09:56:26.834973 | controller |     - '    logger.go:42: 09:55:05 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.834977 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.834982 | controller |     - '    logger.go:42: 09:55:05 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.834986 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.834990 | controller |     - '    logger.go:42: 09:55:05 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:56:26.834994 | controller |     - '    logger.go:42: 09:55:06 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.834999 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.835003 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.835007 | controller |       service'
2026-03-14 09:56:26.835011 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.835015 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.835020 | controller |     - '        ]'
2026-03-14 09:56:26.835024 | controller |     - '    logger.go:42: 09:55:06 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.835034 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.835038 | controller |     - '    logger.go:42: 09:55:06 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.835042 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.835047 | controller |     - '    logger.go:42: 09:55:07 | watcher-cinder/3-remove-cinder | Error from server
2026-03-14 09:56:26.835051 | controller |       (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0"
2026-03-14 09:56:26.835073 | controller |       is waiting to start: ContainerCreating'
2026-03-14 09:56:26.835079 | controller |     - '    logger.go:42: 09:55:07 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:56:26.835083 | controller |     - '    logger.go:42: 09:55:08 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.835087 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.835092 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.835096 | controller |       service'
2026-03-14 09:56:26.835100 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.835104 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.835108 | controller |     - '        ]'
2026-03-14 09:56:26.835113 | controller |     - '    logger.go:42: 09:55:08 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.835117 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.835121 | controller |     - '    logger.go:42: 09:55:08 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.835125 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.835129 | controller |     - '    logger.go:42: 09:55:08 | watcher-cinder/3-remove-cinder | Error from server
2026-03-14 09:56:26.835134 | controller |       (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0"
2026-03-14 09:56:26.835138 | controller |       is waiting to start: ContainerCreating'
2026-03-14 09:56:26.835142 | controller |     - '    logger.go:42: 09:55:08 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:56:26.835146 | controller |     - '    logger.go:42: 09:55:09 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.835150 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.835158 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.835162 | controller |       service'
2026-03-14 09:56:26.835166 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.835171 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.835175 | controller |     - '        ]'
2026-03-14 09:56:26.835179 | controller |     - '    logger.go:42: 09:55:09 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.835183 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.835190 | controller |     - '    logger.go:42: 09:55:09 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.835194 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.835199 | controller |     - '    logger.go:42: 09:55:09 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:56:26.835203 | controller |     - '    logger.go:42: 09:55:10 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.835207 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.835211 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.835216 | controller |       service'
2026-03-14 09:56:26.835220 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.835224 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.835228 | controller |     - '        ]'
2026-03-14 09:56:26.835233 | controller |     - '    logger.go:42: 09:55:10 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.835239 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.835243 | controller |     - '    logger.go:42: 09:55:10 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.835248 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.835252 | controller |     - '    logger.go:42: 09:55:10 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:56:26.835263 | controller |     - '    logger.go:42: 09:55:11 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.835267 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.835277 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.835282 | controller |       service'
2026-03-14 09:56:26.835286 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.835290 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.835294 | controller |     - '        ]'
2026-03-14 09:56:26.835299 | controller |     - '    logger.go:42: 09:55:11 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.835303 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.835307 | controller |     - '    logger.go:42: 09:55:11 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.835312 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.835316 | controller |     - '    logger.go:42: 09:55:11 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']'''
2026-03-14 09:56:26.835320 | controller |     - '    logger.go:42: 09:55:12 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.835325 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.835329 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.835333 | controller |       service'
2026-03-14 09:56:26.835337 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.835342 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.835346 | controller |     - '        ]'
2026-03-14 09:56:26.835350 | controller |     - '    logger.go:42: 09:55:12 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.835354 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.835359 | controller |     - '    logger.go:42: 09:55:12 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.835363 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.835367 | controller |     - '    logger.go:42: 09:55:12 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']'''
2026-03-14 09:56:26.835371 | controller |     - '    logger.go:42: 09:55:14 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.835376 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.835380 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.835384 | controller |       service'
2026-03-14 09:56:26.835388 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.835392 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.835397 | controller |     - '        ]'
2026-03-14 09:56:26.835401 | controller |     - '    logger.go:42: 09:55:14 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.835405 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.835410 | controller |     - '    logger.go:42: 09:55:14 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.835414 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.835418 | controller |     - '    logger.go:42: 09:55:14 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']'''
2026-03-14 09:56:26.835422 | controller |     - '    logger.go:42: 09:55:15 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.835427 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.835433 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.835439 | controller |       service'
2026-03-14 09:56:26.835444 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.835448 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.835452 | controller |     - '        ]'
2026-03-14 09:56:26.835457 | controller |     - '    logger.go:42: 09:55:15 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.835461 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.835465 | controller |     - '    logger.go:42: 09:55:15 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.835469 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.835475 | controller |     - '    logger.go:42: 09:55:15 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']'''
2026-03-14 09:56:26.835479 | controller |     - '    logger.go:42: 09:55:16 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.835484 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.835488 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.835492 | controller |       service'
2026-03-14 09:56:26.835496 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.835500 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.835505 | controller |     - '        ]'
2026-03-14 09:56:26.835509 | controller |     - '    logger.go:42: 09:55:16 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.835513 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.835517 | controller |     - '    logger.go:42: 09:55:16 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.835522 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.835527 | controller |     - '    logger.go:42: 09:55:16 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']'''
2026-03-14 09:56:26.835532 | controller |     - '    logger.go:42: 09:55:17 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.835536 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.835540 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.835544 | controller |       service'
2026-03-14 09:56:26.835549 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.835553 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.835557 | controller |     - '        ]'
2026-03-14 09:56:26.835561 | controller |     - '    logger.go:42: 09:55:17 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.835566 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.835570 | controller |     - '    logger.go:42: 09:55:17 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.835574 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.835578 | controller |     - '    logger.go:42: 09:55:17 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']'''
2026-03-14 09:56:26.835582 | controller |     - '    logger.go:42: 09:55:18 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:56:26.835587 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:56:26.835607 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:56:26.835623 | controller |       service'
2026-03-14 09:56:26.835628 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:56:26.835632 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:56:26.835637 | controller |     - '        ]'
2026-03-14 09:56:26.835641 | controller |     - '    logger.go:42: 09:55:18 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:56:26.835645 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:56:26.835652 | controller |     - '    logger.go:42: 09:55:18 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:56:26.835656 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:56:26.835660 | controller |     - '    logger.go:42: 09:55:19 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']'''
2026-03-14 09:56:26.835665 | controller |     - '    logger.go:42: 09:55:19 | watcher-cinder/3-remove-cinder | test step completed
2026-03-14 09:56:26.835669 | controller |       3-remove-cinder'
2026-03-14 09:56:26.835673 | controller |     - '    logger.go:42: 09:55:19 | watcher-cinder/4-cleanup-watcher | starting test step
2026-03-14 09:56:26.835678 | controller |       4-cleanup-watcher'
2026-03-14 09:56:26.835682 | controller |     - '    logger.go:42: 09:55:26 | watcher-cinder/4-cleanup-watcher | test step completed
2026-03-14 09:56:26.835686 | controller |       4-cleanup-watcher'
2026-03-14 09:56:26.835691 | controller |     - '    logger.go:42: 09:55:26 | watcher-cinder | skipping kubernetes event logging'
2026-03-14 09:56:26.835695 | controller |     - === CONT  kuttl/harness/watcher-api-scaling
2026-03-14 09:56:26.835699 | controller |     - '    logger.go:42: 09:55:26 | watcher-api-scaling | Skipping creation of user-supplied
2026-03-14 09:56:26.835704 | controller |       namespace: watcher-kuttl-default'
2026-03-14 09:56:26.835708 | controller |     - '    logger.go:42: 09:55:26 | watcher-api-scaling/0-cleanup-watcher | starting test
2026-03-14 09:56:26.835712 | controller |       step 0-cleanup-watcher'
2026-03-14 09:56:26.835717 | controller |     - '    logger.go:42: 09:55:26 | watcher-api-scaling/0-cleanup-watcher | test step
2026-03-14 09:56:26.835721 | controller |       completed 0-cleanup-watcher'
2026-03-14 09:56:26.835730 | controller |     - '    logger.go:42: 09:55:26 | watcher-api-scaling/1-deploy-with-defaults | starting
2026-03-14 09:56:26.835735 | controller |       test step 1-deploy-with-defaults'
2026-03-14 09:56:26.835742 | controller |     - '    logger.go:42: 09:55:26 | watcher-api-scaling/1-deploy-with-defaults | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:56:26.835747 | controller |       created'
2026-03-14 09:56:26.835751 | controller |     - '    logger.go:42: 09:55:27 | watcher-api-scaling/1-deploy-with-defaults | running
2026-03-14 09:56:26.835755 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.835760 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.835764 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.835768 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:56:26.835773 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.835777 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.835781 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.835790 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.835795 | controller |       ]'
2026-03-14 09:56:26.835799 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.835804 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.835808 | controller |       == "" ]; then'
2026-03-14 09:56:26.835812 | controller |     - '            exit 0'
2026-03-14 09:56:26.835817 | controller |     - '        fi'
2026-03-14 09:56:26.835821 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.835825 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.835830 | controller |     - '        counter=0'
2026-03-14 09:56:26.835834 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.835838 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.835843 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.835847 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.835851 | controller |     - '            fi'
2026-03-14 09:56:26.835856 | controller |     - '        done'
2026-03-14 09:56:26.835862 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.835866 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.835870 | controller |     - '          exit 1'
2026-03-14 09:56:26.835875 | controller |     - '        else'
2026-03-14 09:56:26.835879 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.835883 | controller |     - '        fi'
2026-03-14 09:56:26.835887 | controller |     - '        ]'
2026-03-14 09:56:26.835892 | controller |     - '    logger.go:42: 09:55:27 | watcher-api-scaling/1-deploy-with-defaults | + oc
2026-03-14 09:56:26.835896 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.835900 | controller |       -c Name -c Type'
2026-03-14 09:56:26.835905 | controller |     - '    logger.go:42: 09:55:27 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:56:26.835909 | controller |       -c ''^watcher'''
2026-03-14 09:56:26.835913 | controller |     - '    logger.go:42: 09:55:29 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.835917 | controller |       0 == 1 '']'''
2026-03-14 09:56:26.835922 | controller |     - '    logger.go:42: 09:55:30 | watcher-api-scaling/1-deploy-with-defaults | running
2026-03-14 09:56:26.835926 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.835930 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.835935 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.835939 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:56:26.835945 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.835950 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.835954 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.835960 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.835964 | controller |       ]'
2026-03-14 09:56:26.835968 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.835973 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.835977 | controller |       == "" ]; then'
2026-03-14 09:56:26.835982 | controller |     - '            exit 0'
2026-03-14 09:56:26.835986 | controller |     - '        fi'
2026-03-14 09:56:26.835990 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.835995 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.835999 | controller |     - '        counter=0'
2026-03-14 09:56:26.836003 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.836007 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.836012 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.836016 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.836020 | controller |     - '            fi'
2026-03-14 09:56:26.836025 | controller |     - '        done'
2026-03-14 09:56:26.836029 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.836033 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.836038 | controller |     - '          exit 1'
2026-03-14 09:56:26.836042 | controller |     - '        else'
2026-03-14 09:56:26.836046 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.836050 | controller |     - '        fi'
2026-03-14 09:56:26.836055 | controller |     - '        ]'
2026-03-14 09:56:26.836059 | controller |     - '    logger.go:42: 09:55:30 | watcher-api-scaling/1-deploy-with-defaults | + oc
2026-03-14 09:56:26.836063 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.836067 | controller |       -c Name -c Type'
2026-03-14 09:56:26.836072 | controller |     - '    logger.go:42: 09:55:30 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:56:26.836080 | controller |       -c ''^watcher'''
2026-03-14 09:56:26.836085 | controller |     - '    logger.go:42: 09:55:32 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.836089 | controller |       0 == 1 '']'''
2026-03-14 09:56:26.836093 | controller |     - '    logger.go:42: 09:55:33 | watcher-api-scaling/1-deploy-with-defaults | running
2026-03-14 09:56:26.836098 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.836102 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.836106 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.836111 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:56:26.836115 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.836119 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.836123 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.836128 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.836132 | controller |       ]'
2026-03-14 09:56:26.836136 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.836141 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.836145 | controller |       == "" ]; then'
2026-03-14 09:56:26.836149 | controller |     - '            exit 0'
2026-03-14 09:56:26.836154 | controller |     - '        fi'
2026-03-14 09:56:26.836158 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.836164 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.836168 | controller |     - '        counter=0'
2026-03-14 09:56:26.836172 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.836177 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.836181 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.836185 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.836190 | controller |     - '            fi'
2026-03-14 09:56:26.836194 | controller |     - '        done'
2026-03-14 09:56:26.836198 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.836203 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.836207 | controller |     - '          exit 1'
2026-03-14 09:56:26.836211 | controller |     - '        else'
2026-03-14 09:56:26.836215 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.836220 | controller |     - '        fi'
2026-03-14 09:56:26.836224 | controller |     - '        ]'
2026-03-14 09:56:26.836228 | controller |     - '    logger.go:42: 09:55:33 | watcher-api-scaling/1-deploy-with-defaults | + oc
2026-03-14 09:56:26.836233 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.836237 | controller |       -c Name -c Type'
2026-03-14 09:56:26.836241 | controller |     - '    logger.go:42: 09:55:33 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:56:26.836246 | controller |       -c ''^watcher'''
2026-03-14 09:56:26.836250 | controller |     - '    logger.go:42: 09:55:36 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.836254 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.836259 | controller |     - '    logger.go:42: 09:55:36 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:56:26.836263 | controller |       watcher'
2026-03-14 09:56:26.836267 | controller |     - '    logger.go:42: 09:55:36 | watcher-api-scaling/1-deploy-with-defaults | ++ awk
2026-03-14 09:56:26.836271 | controller |       ''{print $1}'''
2026-03-14 09:56:26.836276 | controller |     - '    logger.go:42: 09:55:36 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:56:26.836280 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.836284 | controller |       -c Name -c Type -c ID'
2026-03-14 09:56:26.836289 | controller |     - '    logger.go:42: 09:55:38 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=895be62000ea4e65989b573b132a3aa1'
2026-03-14 09:56:26.836295 | controller |     - '    logger.go:42: 09:55:38 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:56:26.836300 | controller |       get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.836312 | controller |     - '    logger.go:42: 09:55:38 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.836320 | controller |       895be62000ea4e65989b573b132a3aa1 == 895be62000ea4e65989b573b132a3aa1 '']'''
2026-03-14 09:56:26.836325 | controller |     - '    logger.go:42: 09:55:38 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:56:26.836330 | controller |       get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.836369 | controller |     - '    logger.go:42: 09:55:38 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.836376 | controller |       -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.836381 | controller |       '']'''
2026-03-14 09:56:26.836386 | controller |     - '    logger.go:42: 09:55:38 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:56:26.836393 | controller |       get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:56:26.836399 | controller |     - '    logger.go:42: 09:55:38 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.836404 | controller |       '''' == '''' '']'''
2026-03-14 09:56:26.836408 | controller |     - '    logger.go:42: 09:55:38 | watcher-api-scaling/1-deploy-with-defaults | + exit
2026-03-14 09:56:26.836416 | controller |       0'
2026-03-14 09:56:26.836421 | controller |     - '    logger.go:42: 09:55:39 | watcher-api-scaling/1-deploy-with-defaults | running
2026-03-14 09:56:26.836426 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.836433 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.836437 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.836442 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:56:26.836450 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.836455 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.836459 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.836467 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.836473 | controller |       ]'
2026-03-14 09:56:26.836477 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.836482 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.836489 | controller |       == "" ]; then'
2026-03-14 09:56:26.836495 | controller |     - '            exit 0'
2026-03-14 09:56:26.836500 | controller |     - '        fi'
2026-03-14 09:56:26.836504 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.836508 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.836516 | controller |     - '        counter=0'
2026-03-14 09:56:26.836522 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.836526 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.836531 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.836535 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.836539 | controller |     - '            fi'
2026-03-14 09:56:26.836544 | controller |     - '        done'
2026-03-14 09:56:26.836548 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.836552 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.836557 | controller |     - '          exit 1'
2026-03-14 09:56:26.836561 | controller |     - '        else'
2026-03-14 09:56:26.836565 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.836572 | controller |     - '        fi'
2026-03-14 09:56:26.836576 | controller |     - '        ]'
2026-03-14 09:56:26.836580 | controller |     - '    logger.go:42: 09:55:39 | watcher-api-scaling/1-deploy-with-defaults | + oc
2026-03-14 09:56:26.836585 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.836589 | controller |       -c Name -c Type'
2026-03-14 09:56:26.836606 | controller |     - '    logger.go:42: 09:55:39 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:56:26.836613 | controller |       -c ''^watcher'''
2026-03-14 09:56:26.836618 | controller |     - '    logger.go:42: 09:55:41 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.836622 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.836626 | controller |     - '    logger.go:42: 09:55:41 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:56:26.836631 | controller |       watcher'
2026-03-14 09:56:26.836635 | controller |     - '    logger.go:42: 09:55:41 | watcher-api-scaling/1-deploy-with-defaults | ++ awk
2026-03-14 09:56:26.836639 | controller |       ''{print $1}'''
2026-03-14 09:56:26.836643 | controller |     - '    logger.go:42: 09:55:41 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:56:26.836648 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.836652 | controller |       -c Name -c Type -c ID'
2026-03-14 09:56:26.836656 | controller |     - '    logger.go:42: 09:55:43 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=895be62000ea4e65989b573b132a3aa1'
2026-03-14 09:56:26.836661 | controller |     - '    logger.go:42: 09:55:43 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:56:26.836665 | controller |       get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.836669 | controller |     - '    logger.go:42: 09:55:43 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.836674 | controller |       895be62000ea4e65989b573b132a3aa1 == 895be62000ea4e65989b573b132a3aa1 '']'''
2026-03-14 09:56:26.836678 | controller |     - '    logger.go:42: 09:55:43 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:56:26.836682 | controller |       get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.836687 | controller |     - '    logger.go:42: 09:55:43 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.836691 | controller |       -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.836695 | controller |       '']'''
2026-03-14 09:56:26.836700 | controller |     - '    logger.go:42: 09:55:43 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:56:26.836704 | controller |       get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:56:26.836708 | controller |     - '    logger.go:42: 09:55:44 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.836712 | controller |       '''' == '''' '']'''
2026-03-14 09:56:26.836717 | controller |     - '    logger.go:42: 09:55:44 | watcher-api-scaling/1-deploy-with-defaults | + exit
2026-03-14 09:56:26.836721 | controller |       0'
2026-03-14 09:56:26.836725 | controller |     - '    logger.go:42: 09:55:45 | watcher-api-scaling/1-deploy-with-defaults | running
2026-03-14 09:56:26.836729 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.836734 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.836738 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.836742 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:56:26.836747 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.836754 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.836758 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.836762 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.836769 | controller |       ]'
2026-03-14 09:56:26.836773 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.836778 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.836782 | controller |       == "" ]; then'
2026-03-14 09:56:26.836786 | controller |     - '            exit 0'
2026-03-14 09:56:26.836791 | controller |     - '        fi'
2026-03-14 09:56:26.836795 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.836799 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.836803 | controller |     - '        counter=0'
2026-03-14 09:56:26.836808 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.836812 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.836816 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.836821 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.836825 | controller |     - '            fi'
2026-03-14 09:56:26.836829 | controller |     - '        done'
2026-03-14 09:56:26.836839 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.836847 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.836852 | controller |     - '          exit 1'
2026-03-14 09:56:26.836856 | controller |     - '        else'
2026-03-14 09:56:26.836860 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.836865 | controller |     - '        fi'
2026-03-14 09:56:26.836869 | controller |     - '        ]'
2026-03-14 09:56:26.836873 | controller |     - '    logger.go:42: 09:55:45 | watcher-api-scaling/1-deploy-with-defaults | + oc
2026-03-14 09:56:26.836878 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.836882 | controller |       -c Name -c Type'
2026-03-14 09:56:26.836886 | controller |     - '    logger.go:42: 09:55:45 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:56:26.836890 | controller |       -c ''^watcher'''
2026-03-14 09:56:26.836895 | controller |     - '    logger.go:42: 09:55:47 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.836899 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.836903 | controller |     - '    logger.go:42: 09:55:47 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:56:26.836907 | controller |       watcher'
2026-03-14 09:56:26.836912 | controller |     - '    logger.go:42: 09:55:47 | watcher-api-scaling/1-deploy-with-defaults | ++ awk
2026-03-14 09:56:26.836916 | controller |       ''{print $1}'''
2026-03-14 09:56:26.836920 | controller |     - '    logger.go:42: 09:55:47 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:56:26.836927 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.836931 | controller |       -c Name -c Type -c ID'
2026-03-14 09:56:26.836937 | controller |     - '    logger.go:42: 09:55:49 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=895be62000ea4e65989b573b132a3aa1'
2026-03-14 09:56:26.836941 | controller |     - '    logger.go:42: 09:55:49 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:56:26.836945 | controller |       get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.836950 | controller |     - '    logger.go:42: 09:55:49 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.836954 | controller |       895be62000ea4e65989b573b132a3aa1 == 895be62000ea4e65989b573b132a3aa1 '']'''
2026-03-14 09:56:26.836958 | controller |     - '    logger.go:42: 09:55:49 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:56:26.836962 | controller |       get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.836966 | controller |     - '    logger.go:42: 09:55:49 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.836971 | controller |       -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.836975 | controller |       '']'''
2026-03-14 09:56:26.836979 | controller |     - '    logger.go:42: 09:55:49 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:56:26.836985 | controller |       get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:56:26.836989 | controller |     - '    logger.go:42: 09:55:49 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.836994 | controller |       '''' == '''' '']'''
2026-03-14 09:56:26.836998 | controller |     - '    logger.go:42: 09:55:49 | watcher-api-scaling/1-deploy-with-defaults | + exit
2026-03-14 09:56:26.837002 | controller |       0'
2026-03-14 09:56:26.837006 | controller |     - '    logger.go:42: 09:55:50 | watcher-api-scaling/1-deploy-with-defaults | running
2026-03-14 09:56:26.837011 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.837015 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.837019 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.837023 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:56:26.837027 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.837032 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.837036 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.837040 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.837044 | controller |       ]'
2026-03-14 09:56:26.837051 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.837055 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.837059 | controller |       == "" ]; then'
2026-03-14 09:56:26.837064 | controller |     - '            exit 0'
2026-03-14 09:56:26.837068 | controller |     - '        fi'
2026-03-14 09:56:26.837072 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.837076 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.837080 | controller |     - '        counter=0'
2026-03-14 09:56:26.837085 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.837089 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.837093 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.837097 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.837102 | controller |     - '            fi'
2026-03-14 09:56:26.837106 | controller |     - '        done'
2026-03-14 09:56:26.837110 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.837114 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.837118 | controller |     - '          exit 1'
2026-03-14 09:56:26.837123 | controller |     - '        else'
2026-03-14 09:56:26.837127 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.837131 | controller |     - '        fi'
2026-03-14 09:56:26.837135 | controller |     - '        ]'
2026-03-14 09:56:26.837140 | controller |     - '    logger.go:42: 09:55:50 | watcher-api-scaling/1-deploy-with-defaults | + oc
2026-03-14 09:56:26.837144 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.837148 | controller |       -c Name -c Type'
2026-03-14 09:56:26.837152 | controller |     - '    logger.go:42: 09:55:50 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:56:26.837156 | controller |       -c ''^watcher'''
2026-03-14 09:56:26.837161 | controller |     - '    logger.go:42: 09:55:53 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.837165 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.837169 | controller |     - '    logger.go:42: 09:55:53 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:56:26.837173 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.837177 | controller |       -c Name -c Type -c ID'
2026-03-14 09:56:26.837182 | controller |     - '    logger.go:42: 09:55:53 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:56:26.837190 | controller |       watcher'
2026-03-14 09:56:26.837196 | controller |     - '    logger.go:42: 09:55:53 | watcher-api-scaling/1-deploy-with-defaults | ++ awk
2026-03-14 09:56:26.837202 | controller |       ''{print $1}'''
2026-03-14 09:56:26.837207 | controller |     - '    logger.go:42: 09:55:55 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=895be62000ea4e65989b573b132a3aa1'
2026-03-14 09:56:26.837213 | controller |     - '    logger.go:42: 09:55:55 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:56:26.837219 | controller |       get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.837223 | controller |     - '    logger.go:42: 09:55:56 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.837227 | controller |       895be62000ea4e65989b573b132a3aa1 == 895be62000ea4e65989b573b132a3aa1 '']'''
2026-03-14 09:56:26.837232 | controller |     - '    logger.go:42: 09:55:56 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:56:26.837236 | controller |       get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.837240 | controller |     - '    logger.go:42: 09:55:56 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.837245 | controller |       -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.837250 | controller |       '']'''
2026-03-14 09:56:26.837256 | controller |     - '    logger.go:42: 09:55:56 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:56:26.837261 | controller |       get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:56:26.837267 | controller |     - '    logger.go:42: 09:55:56 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.837272 | controller |       '''' == '''' '']'''
2026-03-14 09:56:26.837281 | controller |     - '    logger.go:42: 09:55:56 | watcher-api-scaling/1-deploy-with-defaults | + exit
2026-03-14 09:56:26.837285 | controller |       0'
2026-03-14 09:56:26.837290 | controller |     - '    logger.go:42: 09:55:57 | watcher-api-scaling/1-deploy-with-defaults | running
2026-03-14 09:56:26.837294 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:56:26.837298 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.837302 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:56:26.837307 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:56:26.837311 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:56:26.837315 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:56:26.837319 | controller |       == ${SERVICEID} ]'
2026-03-14 09:56:26.837324 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:56:26.837328 | controller |       ]'
2026-03-14 09:56:26.837332 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:56:26.837336 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:56:26.837342 | controller |       == "" ]; then'
2026-03-14 09:56:26.837347 | controller |     - '            exit 0'
2026-03-14 09:56:26.837352 | controller |     - '        fi'
2026-03-14 09:56:26.837356 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:56:26.837372 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:56:26.837377 | controller |     - '        counter=0'
2026-03-14 09:56:26.837381 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:56:26.837386 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:56:26.837390 | controller |     - '                echo ${i}'
2026-03-14 09:56:26.837394 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:56:26.837398 | controller |     - '            fi'
2026-03-14 09:56:26.837403 | controller |     - '        done'
2026-03-14 09:56:26.837407 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:56:26.837413 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:56:26.837417 | controller |     - '          exit 1'
2026-03-14 09:56:26.837422 | controller |     - '        else'
2026-03-14 09:56:26.837426 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:56:26.837430 | controller |     - '        fi'
2026-03-14 09:56:26.837435 | controller |     - '        ]'
2026-03-14 09:56:26.837439 | controller |     - '    logger.go:42: 09:55:57 | watcher-api-scaling/1-deploy-with-defaults | + oc
2026-03-14 09:56:26.837443 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.837447 | controller |       -c Name -c Type'
2026-03-14 09:56:26.837452 | controller |     - '    logger.go:42: 09:55:57 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:56:26.837456 | controller |       -c ''^watcher'''
2026-03-14 09:56:26.837460 | controller |     - '    logger.go:42: 09:55:59 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.837464 | controller |       1 == 1 '']'''
2026-03-14 09:56:26.837469 | controller |     - '    logger.go:42: 09:55:59 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:56:26.837473 | controller |       watcher'
2026-03-14 09:56:26.837477 | controller |     - '    logger.go:42: 09:55:59 | watcher-api-scaling/1-deploy-with-defaults | ++ awk
2026-03-14 09:56:26.837481 | controller |       ''{print $1}'''
2026-03-14 09:56:26.837486 | controller |     - '    logger.go:42: 09:55:59 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:56:26.837490 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:56:26.837496 | controller |       -c Name -c Type -c ID'
2026-03-14 09:56:26.837501 | controller |     - '    logger.go:42: 09:56:01 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=895be62000ea4e65989b573b132a3aa1'
2026-03-14 09:56:26.837505 | controller |     - '    logger.go:42: 09:56:01 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:56:26.837511 | controller |       get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:56:26.837515 | controller |     - '    logger.go:42: 09:56:02 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.837519 | controller |       895be62000ea4e65989b573b132a3aa1 == 895be62000ea4e65989b573b132a3aa1 '']'''
2026-03-14 09:56:26.837524 | controller |     - '    logger.go:42: 09:56:02 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:56:26.837528 | controller |       get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:56:26.837532 | controller |     - '    logger.go:42: 09:56:02 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.837536 | controller |       -n n56dh598hdchcbhbdh5b7h586h588h7h588hbch5c7h6h68h95h664h576h57fh59fh54fh7fh576h57ch588h659hdfh5c5h586hch89h559hd5q
2026-03-14 09:56:26.837541 | controller |       '']'''
2026-03-14 09:56:26.837545 | controller |     - '    logger.go:42: 09:56:02 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:56:26.837549 | controller |       get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:56:26.837554 | controller |     - '    logger.go:42: 09:56:02 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:56:26.837558 | controller |       '''' == '''' '']'''
2026-03-14 09:56:26.837562 | controller |     - '    logger.go:42: 09:56:02 | watcher-api-scaling/1-deploy-with-defaults | + exit
2026-03-14 09:56:26.837566 | controller |       0'
2026-03-14 09:56:26.837571 | controller |     - '    logger.go:42: 09:56:02 | watcher-api-scaling/1-deploy-with-defaults | test
2026-03-14 09:56:26.837575 | controller |       step completed 1-deploy-with-defaults'
2026-03-14 09:56:26.837579 | controller |     - '    logger.go:42: 09:56:02 | watcher-api-scaling/2-scale-up-watcher-api | starting
2026-03-14 09:56:26.837583 | controller |       test step 2-scale-up-watcher-api'
2026-03-14 09:56:26.837588 | controller |     - '    logger.go:42: 09:56:02 | watcher-api-scaling/2-scale-up-watcher-api | running
2026-03-14 09:56:26.837592 | controller |       command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op":
2026-03-14 09:56:26.837610 | controller |       "replace", "path": "/spec/apiServiceTemplate/replicas", "value":3}]'''
2026-03-14 09:56:26.837618 | controller |     - '        ]'
2026-03-14 09:56:26.837622 | controller |     - '    logger.go:42: 09:56:02 | watcher-api-scaling/2-scale-up-watcher-api | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:56:26.837626 | controller |       patched'
2026-03-14 09:56:26.837631 | controller |     - '    logger.go:42: 09:56:13 | watcher-api-scaling/2-scale-up-watcher-api | test
2026-03-14 09:56:26.837635 | controller |       step completed 2-scale-up-watcher-api'
2026-03-14 09:56:26.837639 | controller |     - '    logger.go:42: 09:56:13 | watcher-api-scaling/3-scale-down-watcher-api | starting
2026-03-14 09:56:26.837643 | controller |       test step 3-scale-down-watcher-api'
2026-03-14 09:56:26.837648 | controller |     - '    logger.go:42: 09:56:13 | watcher-api-scaling/3-scale-down-watcher-api | running
2026-03-14 09:56:26.837652 | controller |       command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op":
2026-03-14 09:56:26.837656 | controller |       "replace", "path": "/spec/apiServiceTemplate/replicas", "value":1}]'''
2026-03-14 09:56:26.837660 | controller |     - '        ]'
2026-03-14 09:56:26.837664 | controller |     - '    logger.go:42: 09:56:14 | watcher-api-scaling/3-scale-down-watcher-api | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:56:26.837669 | controller |       patched'
2026-03-14 09:56:26.837675 | controller |     - '    logger.go:42: 09:56:18 | watcher-api-scaling/3-scale-down-watcher-api | test
2026-03-14 09:56:26.837680 | controller |       step completed 3-scale-down-watcher-api'
2026-03-14 09:56:26.837684 | controller |     - '    logger.go:42: 09:56:18 | watcher-api-scaling/4-scale-down-zero-watcher-api
2026-03-14 09:56:26.837688 | controller |       | starting test step 4-scale-down-zero-watcher-api'
2026-03-14 09:56:26.837695 | controller |     - '    logger.go:42: 09:56:18 | watcher-api-scaling/4-scale-down-zero-watcher-api
2026-03-14 09:56:26.837699 | controller |       | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json''
2026-03-14 09:56:26.837703 | controller |       -p=''[{"op": "replace", "path": "/spec/apiServiceTemplate/replicas", "value":0}]'''
2026-03-14 09:56:26.837708 | controller |     - '        ]'
2026-03-14 09:56:26.837712 | controller |     - '    logger.go:42: 09:56:18 | watcher-api-scaling/4-scale-down-zero-watcher-api
2026-03-14 09:56:26.837716 | controller |       | watcher.watcher.openstack.org/watcher-kuttl patched'
2026-03-14 09:56:26.837720 | controller |     - '    logger.go:42: 09:56:19 | watcher-api-scaling/4-scale-down-zero-watcher-api
2026-03-14 09:56:26.837724 | controller |       | test step completed 4-scale-down-zero-watcher-api'
2026-03-14 09:56:26.837729 | controller |     - '    logger.go:42: 09:56:19 | watcher-api-scaling/5-cleanup-watcher | starting test
2026-03-14 09:56:26.837733 | controller |       step 5-cleanup-watcher'
2026-03-14 09:56:26.837737 | controller |     - '    logger.go:42: 09:56:19 | watcher-api-scaling/5-cleanup-watcher | running command:
2026-03-14 09:56:26.837741 | controller |       [sh -c set -ex'
2026-03-14 09:56:26.837745 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.837750 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:56:26.837754 | controller |     - '        ]'
2026-03-14 09:56:26.837758 | controller |     - '    logger.go:42: 09:56:19 | watcher-api-scaling/5-cleanup-watcher | + oc exec
2026-03-14 09:56:26.837762 | controller |       -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.837767 | controller |       -c Type'
2026-03-14 09:56:26.837771 | controller |     - '    logger.go:42: 09:56:19 | watcher-api-scaling/5-cleanup-watcher | ++ grep -c
2026-03-14 09:56:26.837775 | controller |       ''^watcher'''
2026-03-14 09:56:26.837779 | controller |     - '    logger.go:42: 09:56:22 | watcher-api-scaling/5-cleanup-watcher | + ''['' 1
2026-03-14 09:56:26.837783 | controller |       == 0 '']'''
2026-03-14 09:56:26.837788 | controller |     - '    logger.go:42: 09:56:23 | watcher-api-scaling/5-cleanup-watcher | running command:
2026-03-14 09:56:26.837792 | controller |       [sh -c set -ex'
2026-03-14 09:56:26.837796 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:56:26.837800 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:56:26.837806 | controller |     - '        ]'
2026-03-14 09:56:26.837811 | controller |     - '    logger.go:42: 09:56:23 | watcher-api-scaling/5-cleanup-watcher | + oc exec
2026-03-14 09:56:26.837815 | controller |       -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:56:26.837819 | controller |       -c Type'
2026-03-14 09:56:26.837823 | controller |     - '    logger.go:42: 09:56:23 | watcher-api-scaling/5-cleanup-watcher | ++ grep -c
2026-03-14 09:56:26.837827 | controller |       ''^watcher'''
2026-03-14 09:56:26.837832 | controller |     - '    logger.go:42: 09:56:25 | watcher-api-scaling/5-cleanup-watcher | + ''['' 0
2026-03-14 09:56:26.837836 | controller |       == 0 '']'''
2026-03-14 09:56:26.837840 | controller |     - '    logger.go:42: 09:56:25 | watcher-api-scaling/5-cleanup-watcher | test step
2026-03-14 09:56:26.837844 | controller |       completed 5-cleanup-watcher'
2026-03-14 09:56:26.837852 | controller |     - '    logger.go:42: 09:56:25 | watcher-api-scaling | skipping kubernetes event logging'
2026-03-14 09:56:26.837856 | controller |     - === CONT  kuttl/harness/deps
2026-03-14 09:56:26.837861 | controller |     - '    logger.go:42: 09:56:25 | deps | Ignoring infra.yaml as it does not match file
2026-03-14 09:56:26.837865 | controller |       name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$'
2026-03-14 09:56:26.837869 | controller |     - '    logger.go:42: 09:56:25 | deps | Ignoring keystone.yaml as it does not match
2026-03-14 09:56:26.837874 | controller |       file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$'
2026-03-14 09:56:26.837878 | controller |     - '    logger.go:42: 09:56:25 | deps | Ignoring kustomization.yaml as it does not
2026-03-14 09:56:26.837882 | controller |       match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$'
2026-03-14 09:56:26.837886 | controller |     - '    logger.go:42: 09:56:25 | deps | Ignoring namespace.yaml as it does not match
2026-03-14 09:56:26.837891 | controller |       file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$'
2026-03-14 09:56:26.837895 | controller |     - '    logger.go:42: 09:56:25 | deps | Ignoring telemetry.yaml as it does not match
2026-03-14 09:56:26.837899 | controller |       file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$'
2026-03-14 09:56:26.837903 | controller |     - '    logger.go:42: 09:56:25 | deps | Skipping creation of user-supplied namespace:
2026-03-14 09:56:26.837908 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.837912 | controller |     - '    logger.go:42: 09:56:25 | deps | skipping kubernetes event logging'
2026-03-14 09:56:26.837916 | controller |     - === CONT  kuttl/harness/common
2026-03-14 09:56:26.837920 | controller |     - '    logger.go:42: 09:56:25 | common | Ignoring cleanup-assert.yaml as it does not
2026-03-14 09:56:26.837925 | controller |       match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$'
2026-03-14 09:56:26.837929 | controller |     - '    logger.go:42: 09:56:25 | common | Ignoring cleanup-errors.yaml as it does not
2026-03-14 09:56:26.837933 | controller |       match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$'
2026-03-14 09:56:26.837938 | controller |     - '    logger.go:42: 09:56:25 | common | Ignoring cleanup-watcher.yaml as it does
2026-03-14 09:56:26.837942 | controller |       not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$'
2026-03-14 09:56:26.837946 | controller |     - '    logger.go:42: 09:56:25 | common | Ignoring deploy-with-defaults.yaml as it
2026-03-14 09:56:26.837952 | controller |       does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$'
2026-03-14 09:56:26.908372 | controller |     - '    logger.go:42: 09:56:25 | common | Skipping creation of user-supplied namespace:
2026-03-14 09:56:26.908407 | controller |       watcher-kuttl-default'
2026-03-14 09:56:26.908418 | controller |     - '    logger.go:42: 09:56:25 | common | skipping kubernetes event logging'
2026-03-14 09:56:26.908423 | controller |     - === NAME  kuttl
2026-03-14 09:56:26.908428 | controller |     - '    harness.go:406: run tests finished'
2026-03-14 09:56:26.908433 | controller |     - '    harness.go:514: cleaning up'
2026-03-14 09:56:26.908437 | controller |     - '    harness.go:571: removing temp folder: ""'
2026-03-14 09:56:26.908442 | controller |     - '--- FAIL: kuttl (1500.34s)'
2026-03-14 09:56:26.908446 | controller |     - '    --- FAIL: kuttl/harness (0.00s)'
2026-03-14 09:56:26.908451 | controller |     - '        --- FAIL: kuttl/harness/appcred-tests (956.65s)'
2026-03-14 09:56:26.908455 | controller |     - '        --- PASS: kuttl/harness/watcher-notification (36.92s)'
2026-03-14 09:56:26.908462 | controller |     - '        --- PASS: kuttl/harness/watcher-topology (38.12s)'
2026-03-14 09:56:26.908467 | controller |     - '        --- PASS: kuttl/harness/watcher-tls-certs-change (42.17s)'
2026-03-14 09:56:26.908471 | controller |     - '        --- PASS: kuttl/harness/watcher-tls (129.08s)'
2026-03-14 09:56:26.908476 | controller |     - '        --- PASS: kuttl/harness/watcher-rmquser (35.43s)'
2026-03-14 09:56:26.908480 | controller |     - '        --- PASS: kuttl/harness/watcher (74.95s)'
2026-03-14 09:56:26.908484 | controller |     - '        --- PASS: kuttl/harness/watcher-cinder (128.08s)'
2026-03-14 09:56:26.908488 | controller |     - '        --- PASS: kuttl/harness/watcher-api-scaling (58.90s)'
2026-03-14 09:56:26.908493 | controller |     - '        --- PASS: kuttl/harness/deps (0.00s)'
2026-03-14 09:56:26.908497 | controller |     - '        --- PASS: kuttl/harness/common (0.00s)'
2026-03-14 09:56:26.908502 | controller |     - FAIL
2026-03-14 09:56:26.908511 | controller |
2026-03-14 09:56:26.968944 | controller | NO MORE HOSTS LEFT *************************************************************
2026-03-14 09:56:26.969008 | controller |
2026-03-14 09:56:26.969017 | controller | PLAY RECAP *********************************************************************
2026-03-14 09:56:26.969023 | controller | controller                 : ok=2    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0
2026-03-14 09:56:26.969029 | controller |
2026-03-14 09:56:26.969035 | controller | Saturday 14 March 2026  09:56:26 +0000 (0:25:01.918)       0:25:02.285 ********
2026-03-14 09:56:26.969040 | controller | ===============================================================================
2026-03-14 09:56:26.969105 | controller | run kuttl test suite from operator Makefile -------------------------- 1501.92s
2026-03-14 09:56:26.969115 | controller | run_hook : Loop on hooks for pre_kuttl_from_operator -------------------- 0.13s
2026-03-14 09:56:26.969121 | controller | run_hook : Assert single hooks are all mappings ------------------------- 0.10s
2026-03-14 09:56:26.969126 | controller | run_hook : Assert parameters are valid ---------------------------------- 0.06s
2026-03-14 09:56:26.969130 | controller | Run hooks before running kuttl tests ------------------------------------ 0.03s
2026-03-14 09:56:26.969135 | controller | Saturday 14 March 2026  09:56:26 +0000 (0:25:01.918)       0:25:02.284 ********
2026-03-14 09:56:26.969139 | controller | ===============================================================================
2026-03-14 09:56:26.969144 | controller | ansible.builtin.command ---------------------------------------------- 1501.92s
2026-03-14 09:56:26.969148 | controller | run_hook ---------------------------------------------------------------- 0.29s
2026-03-14 09:56:26.969152 | controller | ansible.builtin.include_role -------------------------------------------- 0.03s
2026-03-14 09:56:26.969156 | controller | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2026-03-14 09:56:26.969161 | controller | total ---------------------------------------------------------------- 1502.24s
2026-03-14 09:56:28.007799 | controller | ERROR
2026-03-14 09:56:28.008099 | controller | {
2026-03-14 09:56:28.008159 | controller |   "delta": "0:25:02.815984",
2026-03-14 09:56:28.008201 | controller |   "end": "2026-03-14 09:56:26.974539",
2026-03-14 09:56:28.008239 | controller |   "msg": "non-zero return code",
2026-03-14 09:56:28.008275 | controller |   "rc": 2,
2026-03-14 09:56:28.008310 | controller |   "start": "2026-03-14 09:31:24.158555"
2026-03-14 09:56:28.008344 | controller | }
failure
2026-03-14 09:56:28.659124 | 
2026-03-14 09:56:28.659234 | PLAY RECAP
2026-03-14 09:56:28.659275 | controller | ok: 0 changed: 0 unreachable: 0 failed: 1 skipped: 0 rescued: 0 ignored: 0
2026-03-14 09:56:28.659295 | 
2026-03-14 09:56:28.849376 | RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/kuttl/kuttl-from-operator-run.yaml@main]
2026-03-14 09:56:28.859426 | POST-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/e2e-collect-logs.yml@main]
2026-03-14 09:56:29.536988 | 
2026-03-14 09:56:29.537103 | PLAY [Run ci/playbooks/e2e-collect-logs.yml]
2026-03-14 09:56:29.580543 | 
2026-03-14 09:56:29.580646 | TASK [Filter out host if needed]
2026-03-14 09:56:29.592114 | controller | skipping: end_host conditional evaluated to False, continuing execution for controller
2026-03-14 09:56:29.598078 | 
2026-03-14 09:56:29.598142 | TASK [Filter out host if needed]
2026-03-14 09:56:29.612762 | 
2026-03-14 09:56:29.612836 | TASK [Ensure we have the ci-framework on host]
2026-03-14 09:56:30.212150 | controller | ok
2026-03-14 09:56:30.225463 | 
2026-03-14 09:56:30.225596 | TASK [End host if no basedir]
2026-03-14 09:56:30.239743 | controller | skipping: end_host conditional evaluated to False, continuing execution for controller
2026-03-14 09:56:30.251030 | 
2026-03-14 09:56:30.251146 | TASK [Read base centos-9 scenarios]
2026-03-14 09:56:30.276144 | controller | ok
2026-03-14 09:56:30.369734 | 
2026-03-14 09:56:30.369849 | TASK [cifmw_helpers : Fail if file is not yaml or yml extension]
2026-03-14 09:56:30.404256 | controller | skipping: Conditional result was False
2026-03-14 09:56:30.414738 | 
2026-03-14 09:56:30.414826 | TASK [cifmw_helpers : Check if file is available]
2026-03-14 09:56:30.726977 | controller | ok
2026-03-14 09:56:30.734706 | 
2026-03-14 09:56:30.734793 | TASK [cifmw_helpers : Create temporary directory]
2026-03-14 09:56:30.771185 | controller | skipping: Conditional result was False
2026-03-14 09:56:30.785719 | 
2026-03-14 09:56:30.785853 | TASK [cifmw_helpers : Read vars]
2026-03-14 09:56:30.827837 | 
2026-03-14 09:56:30.828123 | TASK [cifmw_helpers : Create new variable files with content]
2026-03-14 09:56:30.880413 | 
2026-03-14 09:56:30.880655 | TASK [cifmw_helpers : Include vars]
2026-03-14 09:56:30.927659 | 
2026-03-14 09:56:30.927983 | TASK [cifmw_helpers : Remove temporary directory]
2026-03-14 09:56:30.964878 | controller | skipping: Conditional result was False
2026-03-14 09:56:30.992278 | 
2026-03-14 09:56:30.992444 | TASK [run_hook : Assert parameters are valid]
2026-03-14 09:56:31.082298 | controller | ok: All assertions passed
2026-03-14 09:56:31.096063 | 
2026-03-14 09:56:31.096201 | TASK [run_hook : Assert single hooks are all mappings]
2026-03-14 09:56:31.188243 | controller | ok: All assertions passed
2026-03-14 09:56:31.203253 | 
2026-03-14 09:56:31.203403 | LOOP [run_hook : Loop on hooks for pre_logs]
2026-03-14 09:56:31.307417 | 
2026-03-14 09:56:31.307763 | TASK [cifmw_setup : Ensure cifmw_basedir param is set]
2026-03-14 09:56:31.334129 | controller | skipping: Conditional result was False
2026-03-14 09:56:31.343244 | 
2026-03-14 09:56:31.343324 | TASK [Read artifacts parameters dir and set as facts]
2026-03-14 09:56:31.366228 | controller | ok
2026-03-14 09:56:31.393891 | 
2026-03-14 09:56:31.394015 | TASK [cifmw_helpers : Check directory is available]
2026-03-14 09:56:31.696932 | controller | ok
2026-03-14 09:56:31.710606 | 
2026-03-14 09:56:31.710776 | TASK [cifmw_helpers : Find yaml files]
2026-03-14 09:56:32.089406 | controller | ok: All paths examined
2026-03-14 09:56:32.096717 | 
2026-03-14 09:56:32.096780 | TASK [cifmw_helpers : Print available yaml files]
2026-03-14 09:56:32.146602 | controller | ok: Found yaml files to parse: ['/home/zuul/ci-framework-data/artifacts/parameters/zuul-params.yml', '/home/zuul/ci-framework-data/artifacts/parameters/install-yamls-params.yml', '/home/zuul/ci-framework-data/artifacts/parameters/custom-params.yml', '/home/zuul/ci-framework-data/artifacts/parameters/openshift-login-params.yml']
2026-03-14 09:56:32.152741 | 
2026-03-14 09:56:32.152806 | LOOP [cifmw_helpers : Create files on localhost and use include_vars]
2026-03-14 09:56:32.200638 | controller | Output suppressed because no_log was given
2026-03-14 09:56:32.201177 | 
2026-03-14 09:56:32.214076 | controller | Output suppressed because no_log was given
2026-03-14 09:56:32.226031 | controller | Output suppressed because no_log was given
2026-03-14 09:56:32.237728 | controller | Output suppressed because no_log was given
2026-03-14 09:56:32.260345 | controller | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/roles/cifmw_helpers/tasks/include_file.yml
2026-03-14 09:56:32.265022 | controller | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/roles/cifmw_helpers/tasks/include_file.yml
2026-03-14 09:56:32.269697 | controller | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/roles/cifmw_helpers/tasks/include_file.yml
2026-03-14 09:56:32.274427 | controller | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/roles/cifmw_helpers/tasks/include_file.yml
2026-03-14 09:56:32.282712 | 
2026-03-14 09:56:32.282770 | TASK [cifmw_helpers : Fail if file is not yaml or yml extension]
2026-03-14 09:56:32.316994 | controller | skipping: Conditional result was False
2026-03-14 09:56:32.324337 | 
2026-03-14 09:56:32.324413 | TASK [cifmw_helpers : Check if file is available]
2026-03-14 09:56:32.589375 | controller | ok
2026-03-14 09:56:32.598414 | 
2026-03-14 09:56:32.598497 | TASK [cifmw_helpers : Create temporary directory]
2026-03-14 09:56:33.029151 | controller -> localhost | changed
2026-03-14 09:56:33.040369 | 
2026-03-14 09:56:33.040468 | TASK [cifmw_helpers : Read vars]
2026-03-14 09:56:33.416176 | controller | Output suppressed because no_log was given
2026-03-14 09:56:33.429485 | 
2026-03-14 09:56:33.429627 | TASK [cifmw_helpers : Create new variable files with content]
2026-03-14 09:56:34.136189 | controller -> localhost | Output suppressed because no_log was given
2026-03-14 09:56:34.143203 | 
2026-03-14 09:56:34.143281 | TASK [cifmw_helpers : Include vars]
2026-03-14 09:56:34.177212 | controller -> localhost | Output suppressed because no_log was given
2026-03-14 09:56:34.183876 | 
2026-03-14 09:56:34.183939 | TASK [cifmw_helpers : Remove temporary directory]
2026-03-14 09:56:34.583118 | controller -> localhost | changed
2026-03-14 09:56:34.590216 | 
2026-03-14 09:56:34.590285 | TASK [cifmw_helpers : Fail if file is not yaml or yml extension]
2026-03-14 09:56:34.613896 | controller | skipping: Conditional result was False
2026-03-14 09:56:34.620826 | 
2026-03-14 09:56:34.620890 | TASK [cifmw_helpers : Check if file is available]
2026-03-14 09:56:34.963616 | controller | ok
2026-03-14 09:56:34.970958 | 
2026-03-14 09:56:34.971019 | TASK [cifmw_helpers : Create temporary directory]
2026-03-14 09:56:35.237536 | controller -> localhost | changed
2026-03-14 09:56:35.245470 | 
2026-03-14 09:56:35.245567 | TASK [cifmw_helpers : Read vars]
2026-03-14 09:56:35.478562 | controller | Output suppressed because no_log was given
2026-03-14 09:56:35.484487 | 
2026-03-14 09:56:35.484546 | TASK [cifmw_helpers : Create new variable files with content]
2026-03-14 09:56:35.922875 | controller -> localhost | Output suppressed because no_log was given
2026-03-14 09:56:35.930604 | 
2026-03-14 09:56:35.930720 | TASK [cifmw_helpers : Include vars]
2026-03-14 09:56:35.966238 | controller -> localhost | Output suppressed because no_log was given
2026-03-14 09:56:35.973453 | 
2026-03-14 09:56:35.973531 | TASK [cifmw_helpers : Remove temporary directory]
2026-03-14 09:56:36.199066 | controller -> localhost | changed
2026-03-14 09:56:36.206106 | 
2026-03-14 09:56:36.206169 | TASK [cifmw_helpers : Fail if file is not yaml or yml extension]
2026-03-14 09:56:36.239861 | controller | skipping: Conditional result was False
2026-03-14 09:56:36.246931 | 
2026-03-14 09:56:36.247002 | TASK [cifmw_helpers : Check if file is available]
2026-03-14 09:56:36.563895 | controller | ok
2026-03-14 09:56:36.570329 | 
2026-03-14 09:56:36.570388 | TASK [cifmw_helpers : Create temporary directory]
2026-03-14 09:56:36.838018 | controller -> localhost | changed
2026-03-14 09:56:36.855719 | 
2026-03-14 09:56:36.855861 | TASK [cifmw_helpers : Read vars]
2026-03-14 09:56:37.110880 | controller | Output suppressed because no_log was given
2026-03-14 09:56:37.125716 | 
2026-03-14 09:56:37.125859 | TASK [cifmw_helpers : Create new variable files with content]
2026-03-14 09:56:37.639808 | controller -> localhost | Output suppressed because no_log was given
2026-03-14 09:56:37.658194 | 
2026-03-14 09:56:37.658358 | TASK [cifmw_helpers : Include vars]
2026-03-14 09:56:37.697359 | controller -> localhost | Output suppressed because no_log was given
2026-03-14 09:56:37.712935 | 
2026-03-14 09:56:37.713073 | TASK [cifmw_helpers : Remove temporary directory]
2026-03-14 09:56:38.025052 | controller -> localhost | changed
2026-03-14 09:56:38.032563 | 
2026-03-14 09:56:38.032638 | TASK [cifmw_helpers : Fail if file is not yaml or yml extension]
2026-03-14 09:56:38.066872 | controller | skipping: Conditional result was False
2026-03-14 09:56:38.079907 | 
2026-03-14 09:56:38.080052 | TASK [cifmw_helpers : Check if file is available]
2026-03-14 09:56:38.313112 | controller | ok
2026-03-14 09:56:38.328581 | 
2026-03-14 09:56:38.328782 | TASK [cifmw_helpers : Create temporary directory]
2026-03-14 09:56:38.601069 | controller -> localhost | changed
2026-03-14 09:56:38.616956 | 
2026-03-14 09:56:38.617097 | TASK [cifmw_helpers : Read vars]
2026-03-14 09:56:38.863659 | controller | Output suppressed because no_log was given
2026-03-14 09:56:38.879728 | 
2026-03-14 09:56:38.879875 | TASK [cifmw_helpers : Create new variable files with content]
2026-03-14 09:56:39.357666 | controller -> localhost | Output suppressed because no_log was given
2026-03-14 09:56:39.374083 | 
2026-03-14 09:56:39.374225 | TASK [cifmw_helpers : Include vars]
2026-03-14 09:56:39.432783 | controller -> localhost | Output suppressed because no_log was given
2026-03-14 09:56:39.447768 | 
2026-03-14 09:56:39.447901 | TASK [cifmw_helpers : Remove temporary directory]
2026-03-14 09:56:39.749979 | controller -> localhost | changed
2026-03-14 09:56:39.763539 | 
2026-03-14 09:56:39.763612 | TASK [cifmw_setup : Set custom cifmw PATH reusable fact]
2026-03-14 09:56:39.789381 | controller | skipping: Conditional result was False
2026-03-14 09:56:39.796701 | 
2026-03-14 09:56:39.796763 | TASK [cifmw_setup : Set destination folder for the logs]
2026-03-14 09:56:39.844090 | controller | ok
2026-03-14 09:56:39.856995 | 
2026-03-14 09:56:39.857078 | TASK [artifacts : Verify if "success" flag exists after successful tests execution]
2026-03-14 09:56:40.130672 | controller | ok
2026-03-14 09:56:40.145214 | 
2026-03-14 09:56:40.145354 | TASK [artifacts : Ensure we have at least minimal ansible facts]
2026-03-14 09:56:40.875527 | controller | ok
2026-03-14 09:56:40.890040 | 
2026-03-14 09:56:40.890170 | LOOP [artifacts : Ensure base directory exists]
2026-03-14 09:56:41.135787 | controller | ok: "artifacts"
2026-03-14 09:56:41.320768 | controller | ok: "logs"
2026-03-14 09:56:41.338595 | 
2026-03-14 09:56:41.338827 | TASK [artifacts : Load generated hook environment for further usage]
2026-03-14 09:56:41.388730 | controller | ERROR
2026-03-14 09:56:41.389088 | controller | {
2026-03-14 09:56:41.389157 | controller |   "ansible_facts": {},
2026-03-14 09:56:41.389200 | controller |   "ansible_included_var_files": [],
2026-03-14 09:56:41.389237 | controller |   "message": "/home/zuul/ci-framework-data/artifacts directory does not exist"
2026-03-14 09:56:41.389273 | controller | }
2026-03-14 09:56:41.389318 | controller | ERROR: Ignoring Errors
2026-03-14 09:56:41.401866 | 
2026-03-14 09:56:41.402005 | TASK [artifacts : Gather parameter passed to the playbook]
2026-03-14 09:56:42.165574 | controller | changed
2026-03-14 09:56:42.178807 | 
2026-03-14 09:56:42.178940 | TASK [artifacts : Gather facts]
2026-03-14 09:56:42.946524 | controller | changed
2026-03-14 09:56:42.960192 | 
2026-03-14 09:56:42.960325 | TASK [artifacts : Copy some network configuration files]
2026-03-14 09:56:43.550038 | controller | changed
2026-03-14 09:56:43.564103 | 
2026-03-14 09:56:43.564246 | TASK [artifacts : Get installed packages list]
2026-03-14 09:56:44.523205 | controller | ok
2026-03-14 09:56:44.580565 | 
2026-03-14 09:56:44.580645 | TASK [artifacts : Output package list]
2026-03-14 09:56:45.347356 | controller | changed
2026-03-14 09:56:45.372937 | 
2026-03-14 09:56:45.373027 | LOOP [artifacts : Cleanup unnecesary directories]
2026-03-14 09:56:45.752279 | controller | changed: "/home/zuul/ci-framework-data/artifacts/manifests/operator"
2026-03-14 09:56:45.752570 | controller | ok: All items complete
2026-03-14 09:56:45.752611 | 
2026-03-14 09:56:45.789099 | 
2026-03-14 09:56:45.789202 | TASK [os_must_gather : Ensure directories are present]
2026-03-14 09:56:46.057572 | controller | changed
2026-03-14 09:56:46.085115 | 
2026-03-14 09:56:46.085200 | LOOP [os_must_gather : Construct project change list]
2026-03-14 09:56:46.179929 | controller | ok:
2026-03-14 09:56:46.180181 | controller | {
2026-03-14 09:56:46.180234 | controller |   "branch": "main",
2026-03-14 09:56:46.180275 | controller |   "change": "352",
2026-03-14 09:56:46.180311 | controller |   "change_message": "openstack-k8s-operators dependency bump branch: main\n\nAutomated changes by [create-pull-request](https://github.com/peter-evans/create-pull-request) GitHub action",
2026-03-14 09:56:46.180354 | controller |   "change_url": "https://github.com/openstack-k8s-operators/watcher-operator/pull/352",
2026-03-14 09:56:46.180389 | controller |   "commit_id": "0d9467a022a6d6a8b6465cdbc9e4aefc533e796a",
2026-03-14 09:56:46.180423 | controller |   "patchset": "0d9467a022a6d6a8b6465cdbc9e4aefc533e796a",
2026-03-14 09:56:46.180458 | controller |   "project": {
2026-03-14 09:56:46.180491 | controller |     "canonical_hostname": "github.com",
2026-03-14 09:56:46.180525 | controller |     "canonical_name": "github.com/openstack-k8s-operators/watcher-operator",
2026-03-14 09:56:46.180558 | controller |     "name": "openstack-k8s-operators/watcher-operator",
2026-03-14 09:56:46.180592 | controller |     "short_name": "watcher-operator",
2026-03-14 09:56:46.180624 | controller |     "src_dir": "src/github.com/openstack-k8s-operators/watcher-operator"
2026-03-14 09:56:46.180657 | controller |   },
2026-03-14 09:56:46.180728 | controller |   "topic": null
2026-03-14 09:56:46.180767 | controller | }
2026-03-14 09:56:46.250947 | 
2026-03-14 09:56:46.251125 | TASK [os_must_gather : Get git tag for image tagging]
2026-03-14 09:56:46.925624 | controller | skipping: Conditional result was False
2026-03-14 09:56:46.945947 | 
2026-03-14 09:56:46.946062 | TASK [os_must_gather : Debug git show-ref]
2026-03-14 09:56:46.982008 | controller | skipping: Conditional result was False
2026-03-14 09:56:47.022885 | 
2026-03-14 09:56:47.023108 | TASK [os_must_gather : Build openstack-must-gather container]
2026-03-14 09:56:47.059656 | controller | skipping: Conditional result was False
2026-03-14 09:56:47.089628 | 
2026-03-14 09:56:47.089764 | TASK [os_must_gather : Push openstack-must-gather container]
2026-03-14 09:56:47.126012 | controller | skipping: Conditional result was False
2026-03-14 09:56:47.156219 | 
2026-03-14 09:56:47.156352 | TASK [os_must_gather : Set cifmw_os_must_gather_image to new image tag]
2026-03-14 09:56:47.194019 | controller | skipping: Conditional result was False
2026-03-14 09:56:47.224450 | 
2026-03-14 09:56:47.224613 | TASK [os_must_gather : Check for oc command]
2026-03-14 09:56:47.699431 | controller | changed
2026-03-14 09:56:47.725251 | 
2026-03-14 09:56:47.725377 | TASK [os_must_gather : Check if kubeconfig exists]
2026-03-14 09:56:48.045898 | controller | ok
2026-03-14 09:56:48.112387 | 
2026-03-14 09:56:48.112510 | TASK [os_must_gather : Run openstack-must-gather command]
2026-03-14 10:02:02.597309 | controller | changed
2026-03-14 10:02:02.635514 | 
2026-03-14 10:02:02.635612 | TASK [os_must_gather : Find existing os-must-gather directories]
2026-03-14 10:02:02.916750 | controller | ok: All paths examined
2026-03-14 10:02:02.942822 | 
2026-03-14 10:02:02.942950 | TASK [os_must_gather : Create a symlink to newest os-must-gather directory]
2026-03-14 10:02:03.235555 | controller | changed
2026-03-14 10:02:03.304151 | 
2026-03-14 10:02:03.304279 | TASK [os_must_gather : Collect pod usage]
2026-03-14 10:02:03.360253 | controller | ok
2026-03-14 10:02:03.381252 | controller | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/roles/os_must_gather/tasks/get_top.yml
2026-03-14 10:02:03.402913 | 
2026-03-14 10:02:03.402979 | TASK [os_must_gather : Get all containers usage - sort by cpu]
2026-03-14 10:02:03.872625 | controller | error: Metrics API not available
2026-03-14 10:02:04.019266 | controller | ERROR
2026-03-14 10:02:04.019602 | controller | {
2026-03-14 10:02:04.019667 | controller |   "delta": "0:00:00.113030",
2026-03-14 10:02:04.019784 | controller |   "end": "2026-03-14 10:02:03.875100",
2026-03-14 10:02:04.019833 | controller |   "msg": "non-zero return code",
2026-03-14 10:02:04.019872 | controller |   "rc": 1,
2026-03-14 10:02:04.019917 | controller |   "start": "2026-03-14 10:02:03.762070"
2026-03-14 10:02:04.019956 | controller | }
2026-03-14 10:02:04.020006 | controller | ERROR: Ignoring Errors
2026-03-14 10:02:04.068851 | 
2026-03-14 10:02:04.068953 | TASK [os_must_gather : Print message when metrics are not available]
2026-03-14 10:02:04.104896 | controller | skipping: Conditional result was False
2026-03-14 10:02:04.125100 | 
2026-03-14 10:02:04.125186 | TASK [os_must_gather : Get all containers usage - sort by memory]
2026-03-14 10:02:04.781925 | controller | skipping: Conditional result was False
2026-03-14 10:02:04.815110 | 
2026-03-14 10:02:04.815331 | TASK [os_must_gather : Find all namespaces directories]
2026-03-14 10:02:04.854340 | controller | skipping: Conditional result was False
2026-03-14 10:02:04.896080 | 
2026-03-14 10:02:04.896233 | LOOP [os_must_gather : Get resource usage by pods per namespace]
2026-03-14 10:02:04.970001 | 
2026-03-14 10:02:04.970220 | TASK [os_must_gather : Get node resource usage]
2026-03-14 10:02:05.640589 | controller | skipping: Conditional result was False
2026-03-14 10:02:05.677746 | 
2026-03-14 10:02:05.677870 | TASK [artifacts : Create crc logs directory]
2026-03-14 10:02:05.929735 | controller | changed
2026-03-14 10:02:05.955428 | 
2026-03-14 10:02:05.955543 | TASK [artifacts : Ensure controller knows CRC ssh keys]
2026-03-14 10:02:06.583396 | controller | changed
2026-03-14 10:02:06.624206 | 
2026-03-14 10:02:06.624394 | TASK [artifacts : Recognize new keypair]
2026-03-14 10:02:06.902990 | controller | ok
2026-03-14 10:02:06.970322 | 
2026-03-14 10:02:06.970479 | TASK [artifacts : Set fact if new keypair exists]
2026-03-14 10:02:07.015838 | controller | skipping: Conditional result was False
2026-03-14 10:02:07.036667 | 
2026-03-14 10:02:07.036787 | TASK [artifacts : Prepare root ssh accesses]
2026-03-14 10:02:11.084337 | controller | changed
2026-03-14 10:02:11.117925 | 
2026-03-14 10:02:11.118051 | TASK [artifacts : Copy logs from CRC VM]
2026-03-14 10:02:13.583819 | controller | changed
2026-03-14 10:02:13.613656 | 
2026-03-14 10:02:13.613821 | TASK [artifacts : List all of the existing virtual machines]
2026-03-14 10:02:13.650130 | controller | skipping: Conditional result was False
2026-03-14 10:02:13.670925 | 
2026-03-14 10:02:13.671030 | TASK [artifacts : Filter out edpm vm]
2026-03-14 10:02:13.706811 | controller | skipping: Conditional result was False
2026-03-14 10:02:13.728511 | 
2026-03-14 10:02:13.728614 | TASK [artifacts : Slurp zuul inventory file]
2026-03-14 10:02:13.983387 | controller | ok
2026-03-14 10:02:14.008588 | 
2026-03-14 10:02:14.008762 | LOOP [artifacts : Extract Compute and Networker from zuul mapping if any]
2026-03-14 10:02:14.084988 | 
2026-03-14 10:02:14.085187 | LOOP [artifacts : Generate logs on edpm vm {{ item.split('/')[0] }}]
2026-03-14 10:02:14.152029 | 
2026-03-14 10:02:14.152231 | LOOP [artifacts : Copy logs to host machine from {{ item.split('/')[0] }}]
2026-03-14 10:02:14.250098 | 
2026-03-14 10:02:14.250253 | TASK [artifacts : Generate list of ansible logs to collect in home directory]
2026-03-14 10:02:14.470843 | controller | ok: All paths examined
2026-03-14 10:02:14.488840 | 
2026-03-14 10:02:14.488916 | LOOP [artifacts : Copy ansible logs to logs directory]
2026-03-14 10:02:14.865684 | controller | changed:
2026-03-14 10:02:14.865954 | controller | {
2026-03-14 10:02:14.865988 | controller |   "atime": 1773479924.6111803,
2026-03-14 10:02:14.866009 | controller |   "ctime": 1773482186.9070723,
2026-03-14 10:02:14.866027 | controller |   "dev": 64513,
2026-03-14 10:02:14.866045 | controller |   "gid": 1000,
2026-03-14 10:02:14.866062 | controller |   "gr_name": "zuul",
2026-03-14 10:02:14.866078 | controller |   "inode": 1197772,
2026-03-14 10:02:14.866094 | controller |   "isblk": false,
2026-03-14 10:02:14.866111 | controller |   "ischr": false,
2026-03-14 10:02:14.866129 | controller |   "isdir": false,
2026-03-14 10:02:14.866146 | controller |   "isfifo": false,
2026-03-14 10:02:14.866163 | controller |   "isgid": false,
2026-03-14 10:02:14.866178 | controller |   "islnk": false,
2026-03-14 10:02:14.866201 | controller |   "isreg": true,
2026-03-14 10:02:14.866219 | controller |   "issock": false,
2026-03-14 10:02:14.866235 | controller |   "isuid": false,
2026-03-14 10:02:14.866252 | controller |   "mode": "0644",
2026-03-14 10:02:14.866268 | controller |   "mtime": 1773482186.9070723,
2026-03-14 10:02:14.866285 | controller |   "nlink": 1,
2026-03-14 10:02:14.866301 | controller |   "path": "/home/zuul/ansible.log",
2026-03-14 10:02:14.866318 | controller |   "pw_name": "zuul",
2026-03-14 10:02:14.866335 | controller |   "rgrp": true,
2026-03-14 10:02:14.866351 | controller |   "roth": true,
2026-03-14 10:02:14.866368 | controller |   "rusr": true,
2026-03-14 10:02:14.866385 | controller |   "size": 5943955,
2026-03-14 10:02:14.866402 | controller |   "uid": 1000,
2026-03-14 10:02:14.866418 | controller |   "wgrp": false,
2026-03-14 10:02:14.866436 | controller |   "woth": false,
2026-03-14 10:02:14.866454 | controller |   "wusr": true,
2026-03-14 10:02:14.866471 | controller |   "xgrp": false,
2026-03-14 10:02:14.866487 | controller |   "xoth": false,
2026-03-14 10:02:14.866504 | controller |   "xusr": false
2026-03-14 10:02:14.866521 | controller | }
2026-03-14 10:02:14.866549 | 
2026-03-14 10:02:14.895861 | 
2026-03-14 10:02:14.895985 | TASK [artifacts : Ensure we have proper rights on the gathered content]
2026-03-14 10:02:17.025984 | controller | changed
2026-03-14 10:02:17.055584 | 
2026-03-14 10:02:17.055769 | LOOP [artifacts : Mask secrets in yaml log files]
2026-03-14 10:04:57.111808 | controller | changed: "/home/zuul/ci-framework-data/logs"
2026-03-14 10:04:57.827101 | controller | changed: "/home/zuul/ci-framework-data/artifacts"
2026-03-14 10:04:57.897760 | 
2026-03-14 10:04:57.897917 | LOOP [env_op_images : Ensure directory is present]
2026-03-14 10:04:58.162171 | controller | ok: "artifacts"
2026-03-14 10:04:58.162609 | controller | changed: All items complete
2026-03-14 10:04:58.162668 | 
2026-03-14 10:04:58.369624 | controller | ok: "logs"
2026-03-14 10:04:58.415827 | 
2026-03-14 10:04:58.415998 | TASK [env_op_images : Check if OpenStackControlPlane is setup]
2026-03-14 10:04:59.091230 | controller | changed
2026-03-14 10:04:59.132346 | 
2026-03-14 10:04:59.132530 | TASK [env_op_images : Get images from the CSV]
2026-03-14 10:04:59.765087 | controller | skipping: Conditional result was False
2026-03-14 10:04:59.853441 | 
2026-03-14 10:04:59.853656 | TASK [env_op_images : Get the images name]
2026-03-14 10:05:00.476800 | controller | skipping: Conditional result was False
2026-03-14 10:05:00.521301 | 
2026-03-14 10:05:00.521486 | TASK [env_op_images : Extract env variable name and images]
2026-03-14 10:05:00.558464 | controller | skipping: Conditional result was False
2026-03-14 10:05:00.585621 | 
2026-03-14 10:05:00.585747 | TASK [env_op_images : Get all pods from all namespaces to find openstack-operator-index]
2026-03-14 10:05:00.621866 | controller | skipping: Conditional result was False
2026-03-14 10:05:00.657966 | 
2026-03-14 10:05:00.658063 | TASK [env_op_images : Retrieve openstack-operator-index pod]
2026-03-14 10:05:00.694895 | controller | skipping: Conditional result was False
2026-03-14 10:05:00.733789 | 
2026-03-14 10:05:00.733943 | TASK [env_op_images : Get all the pods in openstack-operator namespace]
2026-03-14 10:05:00.770566 | controller | skipping: Conditional result was False
2026-03-14 10:05:00.805968 | 
2026-03-14 10:05:00.806089 | TASK [env_op_images : Get operator images and pods]
2026-03-14 10:05:00.842060 | controller | skipping: Conditional result was False
2026-03-14 10:05:00.878532 | 
2026-03-14 10:05:00.878636 | LOOP [env_op_images : Add operator images to the dictionary]
2026-03-14 10:05:00.956336 | 
2026-03-14 10:05:00.956529 | TASK [env_op_images : Write images to file]
2026-03-14 10:05:00.993238 | controller | skipping: Conditional result was False
2026-03-14 10:05:01.086459 | 
2026-03-14 10:05:01.086580 | TASK [cifmw_setup : Create a versioned log folder]
2026-03-14 10:05:01.334611 | controller | changed
2026-03-14 10:05:01.361701 | 
2026-03-14 10:05:01.361803 | TASK [cifmw_setup : Return a list of log files in home directory]
2026-03-14 10:05:01.623597 | controller | ok: All paths examined
2026-03-14 10:05:01.647673 | 
2026-03-14 10:05:01.647791 | TASK [cifmw_setup : Ensure ansible facts cache exists]
2026-03-14 10:05:01.902414 | controller | ok
2026-03-14 10:05:01.926733 | 
2026-03-14 10:05:01.926838 | LOOP [cifmw_setup : Copy logs to proper location]
2026-03-14 10:05:02.328594 | controller | changed:
2026-03-14 10:05:02.328902 | controller | {
2026-03-14 10:05:02.328958 | controller |   "atime": 1773482534.7419522,
2026-03-14 10:05:02.328998 | controller |   "ctime": 1773482186.9070723,
2026-03-14 10:05:02.329033 | controller |   "dev": 64513,
2026-03-14 10:05:02.329067 | controller |   "gid": 1000,
2026-03-14 10:05:02.329235 | controller |   "gr_name": "zuul",
2026-03-14 10:05:02.329289 | controller |   "inode": 1197772,
2026-03-14 10:05:02.329326 | controller |   "isblk": false,
2026-03-14 10:05:02.329361 | controller |   "ischr": false,
2026-03-14 10:05:02.329396 | controller |   "isdir": false,
2026-03-14 10:05:02.329431 | controller |   "isfifo": false,
2026-03-14 10:05:02.329464 | controller |   "isgid": false,
2026-03-14 10:05:02.329496 | controller |   "islnk": false,
2026-03-14 10:05:02.329528 | controller |   "isreg": true,
2026-03-14 10:05:02.329560 | controller |   "issock": false,
2026-03-14 10:05:02.329592 | controller |   "isuid": false,
2026-03-14 10:05:02.329625 | controller |   "mode": "0644",
2026-03-14 10:05:02.329658 | controller |   "mtime": 1773482186.9070723,
2026-03-14 10:05:02.329726 | controller |   "nlink": 1,
2026-03-14 10:05:02.329762 | controller |   "path": "/home/zuul/ansible.log",
2026-03-14 10:05:02.329794 | controller |   "pw_name": "zuul",
2026-03-14 10:05:02.329825 | controller |   "rgrp": true,
2026-03-14 10:05:02.329857 | controller |   "roth": true,
2026-03-14 10:05:02.329889 | controller |   "rusr": true,
2026-03-14 10:05:02.329921 | controller |   "size": 5943955,
2026-03-14 10:05:02.329953 | controller |   "uid": 1000,
2026-03-14 10:05:02.329984 | controller |   "wgrp": false,
2026-03-14 10:05:02.330015 | controller |   "woth": false,
2026-03-14 10:05:02.330047 | controller |   "wusr": true,
2026-03-14 10:05:02.330078 | controller |   "xgrp": false,
2026-03-14 10:05:02.330110 | controller |   "xoth": false,
2026-03-14 10:05:02.330141 | controller |   "xusr": false
2026-03-14 10:05:02.330173 | controller | }
2026-03-14 10:05:02.330260 | 
2026-03-14 10:05:02.373868 | 
2026-03-14 10:05:02.374080 | LOOP [cifmw_setup : Remove original log from home directory]
2026-03-14 10:05:02.653026 | controller | changed:
2026-03-14 10:05:02.653303 | controller | {
2026-03-14 10:05:02.653356 | controller |   "atime": 1773482534.7419522,
2026-03-14 10:05:02.653396 | controller |   "ctime": 1773482186.9070723,
2026-03-14 10:05:02.653432 | controller |   "dev": 64513,
2026-03-14 10:05:02.653468 | controller |   "gid": 1000,
2026-03-14 10:05:02.653590 | controller |   "gr_name": "zuul",
2026-03-14 10:05:02.653735 | controller |   "inode": 1197772,
2026-03-14 10:05:02.653781 | controller |   "isblk": false,
2026-03-14 10:05:02.653815 | controller |   "ischr": false,
2026-03-14 10:05:02.653862 | controller |   "isdir": false,
2026-03-14 10:05:02.653896 | controller |   "isfifo": false,
2026-03-14 10:05:02.653929 | controller |   "isgid": false,
2026-03-14 10:05:02.653960 | controller |   "islnk": false,
2026-03-14 10:05:02.653992 | controller |   "isreg": true,
2026-03-14 10:05:02.654024 | controller |   "issock": false,
2026-03-14 10:05:02.654056 | controller |   "isuid": false,
2026-03-14 10:05:02.654087 | controller |   "mode": "0644",
2026-03-14 10:05:02.654119 | controller |   "mtime": 1773482186.9070723,
2026-03-14 10:05:02.654152 | controller |   "nlink": 1,
2026-03-14 10:05:02.654185 | controller |   "path": "/home/zuul/ansible.log",
2026-03-14 10:05:02.654218 | controller |   "pw_name": "zuul",
2026-03-14 10:05:02.654251 | controller |   "rgrp": true,
2026-03-14 10:05:02.654283 | controller |   "roth": true,
2026-03-14 10:05:02.654316 | controller |   "rusr": true,
2026-03-14 10:05:02.654349 | controller |   "size": 5943955,
2026-03-14 10:05:02.654385 | controller |   "uid": 1000,
2026-03-14 10:05:02.654419 | controller |   "wgrp": false,
2026-03-14 10:05:02.654454 | controller |   "woth": false,
2026-03-14 10:05:02.654488 | controller |   "wusr": true,
2026-03-14 10:05:02.654522 | controller |   "xgrp": false,
2026-03-14 10:05:02.654555 | controller |   "xoth": false,
2026-03-14 10:05:02.654588 | controller |   "xusr": false
2026-03-14 10:05:02.654621 | controller | }
2026-03-14 10:05:02.701248 | 
2026-03-14 10:05:02.701531 | TASK [cifmw_setup : Copy facts to dated directory]
2026-03-14 10:05:03.105749 | controller | changed
2026-03-14 10:05:03.134465 | 
2026-03-14 10:05:03.134609 | TASK [run_hook : Assert parameters are valid]
2026-03-14 10:05:03.415262 | controller | ok: All assertions passed
2026-03-14 10:05:03.433344 | 
2026-03-14 10:05:03.433409 | TASK [run_hook : Assert single hooks are all mappings]
2026-03-14 10:05:03.589450 | controller | ok: All assertions passed
2026-03-14 10:05:03.616017 | 
2026-03-14 10:05:03.616109 | LOOP [run_hook : Loop on hooks for post_logs]
2026-03-14 10:05:03.885343 | 
2026-03-14 10:05:03.885547 | PLAY [Run ci/playbooks/e2e-collect-logs.yml on CRC host]
2026-03-14 10:05:03.904516 | 
2026-03-14 10:05:03.904588 | TASK [Get kubelet journalctl logs]
2026-03-14 10:05:04.945021 | crc | Output suppressed because no_log was given
2026-03-14 10:05:04.960662 | 
2026-03-14 10:05:04.960894 | TASK [Compress logs bigger than 2MB]
2026-03-14 10:05:05.531160 | crc | changed
2026-03-14 10:05:05.545262 | 
2026-03-14 10:05:05.545385 | TASK [Copy files from workspace on node]
2026-03-14 10:05:05.577828 | crc | ok
2026-03-14 10:05:05.613287 | 
2026-03-14 10:05:05.613376 | TASK [fetch-output : Set log path for multiple nodes]
2026-03-14 10:05:05.668307 | crc | ok
2026-03-14 10:05:05.677891 | 
2026-03-14 10:05:05.677972 | TASK [fetch-output : Set log path for single node]
2026-03-14 10:05:05.713392 | crc | skipping: Conditional result was False
2026-03-14 10:05:05.731867 | 
2026-03-14 10:05:05.732055 | LOOP [fetch-output : Ensure local output dirs]
2026-03-14 10:05:06.017876 | crc -> localhost | changed: "/var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/logs/crc"
2026-03-14 10:05:06.018137 | 
2026-03-14 10:05:06.230541 | crc -> localhost | changed: "/var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/artifacts"
2026-03-14 10:05:06.480724 | crc -> localhost | changed: "/var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/docs"
2026-03-14 10:05:06.494956 | 
2026-03-14 10:05:06.495063 | LOOP [fetch-output : Collect logs, artifacts and docs]
2026-03-14 10:05:07.248737 | crc | changed: >f+++++++++ kubelet.log.gz
2026-03-14 10:05:07.819037 | crc | changed: .d..t...... ./
2026-03-14 10:05:08.404404 | crc | changed: .d..t...... ./
2026-03-14 10:05:08.481519 | 
2026-03-14 10:05:08.481656 | PLAY RECAP
2026-03-14 10:05:08.481756 | controller | ok: 75 changed: 34 unreachable: 0 failed: 0 skipped: 39 rescued: 0 ignored: 2
2026-03-14 10:05:08.481803 | crc | ok: 5 changed: 4 unreachable: 0 failed: 0 skipped: 1 rescued: 0 ignored: 0
2026-03-14 10:05:08.481849 | 
2026-03-14 10:05:08.674369 | POST-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/e2e-collect-logs.yml@main]
2026-03-14 10:05:08.684383 | POST-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml@main]
2026-03-14 10:05:09.313184 | 
2026-03-14 10:05:09.313298 | PLAY [Run ci/playbooks/collect-logs.yml]
2026-03-14 10:05:09.430260 | 
2026-03-14 10:05:09.430430 | TASK [Filter out host if needed]
2026-03-14 10:05:09.459674 | controller | skipping: end_host conditional evaluated to False, continuing execution for controller
2026-03-14 10:05:09.467374 | 
2026-03-14 10:05:09.467471 | TASK [Filter out host if needed]
2026-03-14 10:05:09.500403 | 
2026-03-14 10:05:09.500522 | TASK [Ensure file is present]
2026-03-14 10:05:09.987599 | controller | ok
2026-03-14 10:05:10.021547 | 
2026-03-14 10:05:10.021693 | TASK [Manage molecule report file]
2026-03-14 10:05:10.712168 | controller | skipping: Conditional result was False
2026-03-14 10:05:10.769215 | 
2026-03-14 10:05:10.769370 | TASK [Check if we get ci-framework-data basedir]
2026-03-14 10:05:11.073003 | controller | ok
2026-03-14 10:05:11.105219 | 
2026-03-14 10:05:11.105339 | TASK [Create ci-framework-data log directory for zuul]
2026-03-14 10:05:11.566744 | controller | changed
2026-03-14 10:05:11.599961 | 
2026-03-14 10:05:11.600104 | TASK [Copy ci-framework interesting files]
2026-03-14 10:05:12.240691 | controller | cp: cannot stat '/home/zuul/ci-framework-data/tests': No such file or directory
2026-03-14 10:05:12.327277 | controller | changed
2026-03-14 10:05:12.366374 | 
2026-03-14 10:05:12.366522 | TASK [Get SELinux listing]
2026-03-14 10:05:13.057971 | controller | changed
2026-03-14 10:05:13.110331 | 
2026-03-14 10:05:13.110440 | TASK [Generate log index]
2026-03-14 10:05:13.893300 | controller | changed
2026-03-14 10:05:13.926367 | 
2026-03-14 10:05:13.926447 | TASK [Get some env related data]
2026-03-14 10:05:14.673638 | controller | /home/zuul/.local/bin/ansible
2026-03-14 10:05:15.611662 | controller | changed
2026-03-14 10:05:15.663926 | 
2026-03-14 10:05:15.664093 | TASK [Generate list of logs to collect in home directory]
2026-03-14 10:05:16.012353 | controller | ok: All paths examined
2026-03-14 10:05:16.035144 | 
2026-03-14 10:05:16.035219 | LOOP [Copy logs from home directory]
2026-03-14 10:05:16.073950 | 
2026-03-14 10:05:16.074171 | TASK [Copy crio stats log file]
2026-03-14 10:05:16.098371 | controller | skipping: Conditional result was False
2026-03-14 10:05:16.132293 | 
2026-03-14 10:05:16.132421 | TASK [Get SELinux related data]
2026-03-14 10:05:16.514753 | controller | <no matches>
2026-03-14 10:05:16.822179 | controller | ERROR
2026-03-14 10:05:16.822445 | controller | {
2026-03-14 10:05:16.822508 | controller |   "delta": "0:00:00.009127",
2026-03-14 10:05:16.822551 | controller |   "end": "2026-03-14 10:05:16.515313",
2026-03-14 10:05:16.822589 | controller |   "msg": "non-zero return code",
2026-03-14 10:05:16.822625 | controller |   "rc": 1,
2026-03-14 10:05:16.822661 | controller |   "start": "2026-03-14 10:05:16.506186"
2026-03-14 10:05:16.822741 | controller | }
2026-03-14 10:05:16.822793 | controller | ERROR: Ignoring Errors
2026-03-14 10:05:16.874006 | 
2026-03-14 10:05:16.874122 | TASK [Create system configuration directory]
2026-03-14 10:05:17.169133 | controller | changed
2026-03-14 10:05:17.202848 | 
2026-03-14 10:05:17.202986 | TASK [Get some of the system configurations]
2026-03-14 10:05:17.883007 | controller | changed
2026-03-14 10:05:17.934126 | 
2026-03-14 10:05:17.934304 | TASK [Copy generated documentation if available]
2026-03-14 10:05:17.963370 | controller | skipping: Conditional result was False
2026-03-14 10:05:17.999140 | 
2026-03-14 10:05:17.999302 | TASK [Copy generated AsciiDoc documentation if available]
2026-03-14 10:05:18.025504 | controller | skipping: Conditional result was False
2026-03-14 10:05:18.080084 | 
2026-03-14 10:05:18.080224 | TASK [Compress logs bigger than 2MB]
2026-03-14 10:05:36.324311 | controller | changed
2026-03-14 10:05:36.375013 | 
2026-03-14 10:05:36.375128 | TASK [Copy files from workspace on node]
2026-03-14 10:05:36.424640 | controller | ok
2026-03-14 10:05:36.487198 | 
2026-03-14 10:05:36.487312 | TASK [fetch-output : Set log path for multiple nodes]
2026-03-14 10:05:36.555538 | controller | ok
2026-03-14 10:05:36.612144 | 
2026-03-14 10:05:36.612268 | TASK [fetch-output : Set log path for single node]
2026-03-14 10:05:36.637496 | controller | skipping: Conditional result was False
2026-03-14 10:05:36.662944 | 
2026-03-14 10:05:36.663032 | LOOP [fetch-output : Ensure local output dirs]
2026-03-14 10:05:36.915087 | controller -> localhost | changed: "/var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/logs/controller"
2026-03-14 10:05:36.915323 | 
2026-03-14 10:05:37.151501 | controller -> localhost | ok: "/var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/artifacts"
2026-03-14 10:05:37.365392 | controller -> localhost | ok: "/var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/docs"
2026-03-14 10:05:37.402703 | 
2026-03-14 10:05:37.402875 | LOOP [fetch-output : Collect logs, artifacts and docs]
2026-03-14 10:05:41.295554 | controller | changed:
2026-03-14 10:05:41.295825 | controller | .d..t...... ./
2026-03-14 10:05:41.295864 | controller | >f+++++++++ README.html
2026-03-14 10:05:41.295892 | controller | >f+++++++++ dmesg.log
2026-03-14 10:05:41.295918 | controller | >f+++++++++ installed-pkgs.log
2026-03-14 10:05:41.295943 | controller | >f+++++++++ python.log
2026-03-14 10:05:41.295967 | controller | >f+++++++++ registries.conf
2026-03-14 10:05:41.295991 | controller | >f+++++++++ selinux-denials.log
2026-03-14 10:05:41.296014 | controller | >f+++++++++ selinux-listing.log
2026-03-14 10:05:41.296040 | controller | cd+++++++++ ci-framework-data/
2026-03-14 10:05:41.296064 | controller | cd+++++++++ ci-framework-data/artifacts/
2026-03-14 10:05:41.296088 | controller | >f+++++++++ ci-framework-data/artifacts/ansible-facts.yml
2026-03-14 10:05:41.296112 | controller | >f+++++++++ ci-framework-data/artifacts/ansible-vars.yml
2026-03-14 10:05:41.296136 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_000_check_for_oc.sh
2026-03-14 10:05:41.296160 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_000_copy_logs_from_crc.sh
2026-03-14 10:05:41.296184 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_000_prepare_root_ssh.sh
2026-03-14 10:05:41.296208 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_000_run_download.sh
2026-03-14 10:05:41.296231 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_000_run_openstack_must_gather.sh
2026-03-14 10:05:41.296255 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_001_run_hook_without_retry.sh
2026-03-14 10:05:41.296278 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_002_fetch_openshift.sh
2026-03-14 10:05:41.296301 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_003_run_openstack.sh
2026-03-14 10:05:41.296324 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_004_run_hook_without_retry_deploy.sh
2026-03-14 10:05:41.296347 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_005_stop_watcher.sh
2026-03-14 10:05:41.296370 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_006_install_watcher_operator_in.sh
2026-03-14 10:05:41.296393 | controller | >f+++++++++ ci-framework-data/artifacts/hosts
2026-03-14 10:05:41.296416 | controller | >f+++++++++ ci-framework-data/artifacts/install_yamls.sh
2026-03-14 10:05:41.296439 | controller | >f+++++++++ ci-framework-data/artifacts/installed-packages.yml
2026-03-14 10:05:41.296462 | controller | >f+++++++++ ci-framework-data/artifacts/ip-network.txt
2026-03-14 10:05:41.296485 | controller | >f+++++++++ ci-framework-data/artifacts/resolv.conf
2026-03-14 10:05:41.296507 | controller | >f+++++++++ ci-framework-data/artifacts/zuul_inventory.yml
2026-03-14 10:05:41.296530 | controller | cd+++++++++ ci-framework-data/artifacts/NetworkManager/
2026-03-14 10:05:41.296562 | controller | >f+++++++++ ci-framework-data/artifacts/NetworkManager/ci-private-network.nmconnection
2026-03-14 10:05:41.296586 | controller | >f+++++++++ ci-framework-data/artifacts/NetworkManager/ens3.nmconnection
2026-03-14 10:05:41.296609 | controller | cd+++++++++ ci-framework-data/artifacts/ansible_facts.2026-03-14_10-05/
2026-03-14 10:05:41.296632 | controller | cd+++++++++ ci-framework-data/artifacts/ansible_facts.2026-03-14_10-05/ansible_facts_cache/
2026-03-14 10:05:41.296655 | controller | >f+++++++++ ci-framework-data/artifacts/ansible_facts.2026-03-14_10-05/ansible_facts_cache/controller
2026-03-14 10:05:41.296698 | controller | >f+++++++++ ci-framework-data/artifacts/ansible_facts.2026-03-14_10-05/ansible_facts_cache/localhost
2026-03-14 10:05:41.296729 | controller | cd+++++++++ ci-framework-data/artifacts/ci-env/
2026-03-14 10:05:41.296753 | controller | >f+++++++++ ci-framework-data/artifacts/ci-env/networking-info.yml
2026-03-14 10:05:41.296777 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/
2026-03-14 10:05:41.296803 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/
2026-03-14 10:05:41.296831 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/namespace.yaml
2026-03-14 10:05:41.296863 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/cert-manager/
2026-03-14 10:05:41.296890 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/cert-manager/op/
2026-03-14 10:05:41.296920 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/cert-manager/op/operatorgroup.yaml
2026-03-14 10:05:41.296946 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/cert-manager/op/subscription.yaml
2026-03-14 10:05:41.296970 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/metallb-system/
2026-03-14 10:05:41.296994 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/namespace.yaml
2026-03-14 10:05:41.297018 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/
2026-03-14 10:05:41.297041 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/
2026-03-14 10:05:41.297064 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/bgpadvertisement.yaml
2026-03-14 10:05:41.297087 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/bgpextras.yaml
2026-03-14 10:05:41.297109 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/bgppeers.yaml
2026-03-14 10:05:41.297138 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/deploy_operator.yaml
2026-03-14 10:05:41.297168 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/ipaddresspools.yaml
2026-03-14 10:05:41.297199 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/l2advertisement.yaml
2026-03-14 10:05:41.297228 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openshift-nmstate/
2026-03-14 10:05:41.297258 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openshift-nmstate/namespace.yaml
2026-03-14 10:05:41.297287 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openshift-nmstate/nmstate/
2026-03-14 10:05:41.297316 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openshift-nmstate/nmstate/cr/
2026-03-14 10:05:41.297344 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openshift-nmstate/nmstate/cr/deploy_operator.yaml
2026-03-14 10:05:41.297373 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/
2026-03-14 10:05:41.297401 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/namespace.yaml
2026-03-14 10:05:41.297430 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/metallb/
2026-03-14 10:05:41.297545 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/metallb/op/
2026-03-14 10:05:41.297578 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/metallb/op/operatorgroup.yaml
2026-03-14 10:05:41.297603 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/metallb/op/subscription.yaml
2026-03-14 10:05:41.297626 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/nmstate/
2026-03-14 10:05:41.297649 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/nmstate/op/
2026-03-14 10:05:41.297672 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/nmstate/op/operatorgroup.yaml
2026-03-14 10:05:41.297718 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/nmstate/op/subscription.yaml
2026-03-14 10:05:41.297742 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/openstack/
2026-03-14 10:05:41.297765 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/openstack/op/
2026-03-14 10:05:41.297786 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/openstack/op/catalogsource.yaml
2026-03-14 10:05:41.297809 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/openstack/op/operatorgroup.yaml
2026-03-14 10:05:41.297831 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/openstack/op/subscription.yaml
2026-03-14 10:05:41.297853 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/
2026-03-14 10:05:41.297875 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/namespace.yaml
2026-03-14 10:05:41.297898 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/cr/
2026-03-14 10:05:41.297921 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/
2026-03-14 10:05:41.297944 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/
2026-03-14 10:05:41.297966 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/ctlplane.yaml
2026-03-14 10:05:41.297988 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/datacentre.yaml
2026-03-14 10:05:41.298011 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/designate.yaml
2026-03-14 10:05:41.298033 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/designateext.yaml
2026-03-14 10:05:41.298055 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/internalapi.yaml
2026-03-14 10:05:41.298078 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/octavia.yaml
2026-03-14 10:05:41.298100 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/storage.yaml
2026-03-14 10:05:41.298122 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/storagemgmt.yaml
2026-03-14 10:05:41.298144 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/tenant.yaml
2026-03-14 10:05:41.298166 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/nncp/
2026-03-14 10:05:41.298189 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/nncp/cr/
2026-03-14 10:05:41.298212 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/nncp/cr/crc_nncp.yaml
2026-03-14 10:05:41.298234 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/openstack/
2026-03-14 10:05:41.298257 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/openstack/cr/
2026-03-14 10:05:41.298279 | controller | cd+++++++++ ci-framework-data/artifacts/parameters/
2026-03-14 10:05:41.298302 | controller | >f+++++++++ ci-framework-data/artifacts/parameters/custom-params.yml
2026-03-14 10:05:41.298330 | controller | >f+++++++++ ci-framework-data/artifacts/parameters/install-yamls-params.yml
2026-03-14 10:05:41.298354 | controller | >f+++++++++ ci-framework-data/artifacts/parameters/openshift-login-params.yml
2026-03-14 10:05:41.298376 | controller | >f+++++++++ ci-framework-data/artifacts/parameters/zuul-params.yml
2026-03-14 10:05:41.298398 | controller | cd+++++++++ ci-framework-data/artifacts/repositories/
2026-03-14 10:05:41.298420 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/delorean-antelope-testing.repo
2026-03-14 10:05:41.298442 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/delorean.repo
2026-03-14 10:05:41.298464 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/delorean.repo.md5
2026-03-14 10:05:41.298486 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/repo-setup-centos-appstream.repo
2026-03-14 10:05:41.298509 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/repo-setup-centos-baseos.repo
2026-03-14 10:05:41.298531 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/repo-setup-centos-highavailability.repo
2026-03-14 10:05:41.298553 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/repo-setup-centos-powertools.repo
2026-03-14 10:05:41.298576 | controller | cd+++++++++ ci-framework-data/artifacts/roles/
2026-03-14 10:05:41.298600 | controller | cd+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/
2026-03-14 10:05:41.298623 | controller | cd+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/
2026-03-14 10:05:41.298646 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_all.yml
2026-03-14 10:05:41.298669 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_attach_default_interface.yml
2026-03-14 10:05:41.298718 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_attach_default_interface_cleanup.yml
2026-03-14 10:05:41.298743 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican.yml
2026-03-14 10:05:41.298766 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_cleanup.yml
2026-03-14 10:05:41.298789 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_deploy.yml
2026-03-14 10:05:41.298812 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_deploy_cleanup.yml
2026-03-14 10:05:41.298834 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_deploy_prep.yml
2026-03-14 10:05:41.298857 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_deploy_validate.yml
2026-03-14 10:05:41.298879 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_kuttl.yml
2026-03-14 10:05:41.298901 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_kuttl_run.yml
2026-03-14 10:05:41.298923 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_prep.yml
2026-03-14 10:05:41.298945 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_baremetal.yml
2026-03-14 10:05:41.298967 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_baremetal_cleanup.yml
2026-03-14 10:05:41.298990 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_baremetal_prep.yml
2026-03-14 10:05:41.299012 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas.yml
2026-03-14 10:05:41.299036 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_baremetal_net_nad.yml
2026-03-14 10:05:41.299077 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_baremetal_net_nad_cleanup.yml
2026-03-14 10:05:41.299107 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_cleanup.yml
2026-03-14 10:05:41.299131 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_crc_attach_network.yml
2026-03-14 10:05:41.299153 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_crc_attach_network_cleanup.yml
2026-03-14 10:05:41.299176 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_crc_baremetal_bridge.yml
2026-03-14 10:05:41.299198 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_crc_baremetal_bridge_cleanup.yml
2026-03-14 10:05:41.299221 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_generate_nodes_yaml.yml
2026-03-14 10:05:41.299243 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_metallb.yml
2026-03-14 10:05:41.299266 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_metallb_cleanup.yml
2026-03-14 10:05:41.299289 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_network.yml
2026-03-14 10:05:41.299311 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_network_cleanup.yml
2026-03-14 10:05:41.299334 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_route_crc_and_crc_bmaas_networks.yml
2026-03-14 10:05:41.299357 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_route_crc_and_crc_bmaas_networks_cleanup.yml
2026-03-14 10:05:41.299379 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_sushy_emulator.yml
2026-03-14 10:05:41.299402 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_sushy_emulator_cleanup.yml
2026-03-14 10:05:41.299424 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_sushy_emulator_wait.yml
2026-03-14 10:05:41.299447 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_virtual_bms.yml
2026-03-14 10:05:41.299469 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_virtual_bms_cleanup.yml
2026-03-14 10:05:41.299492 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ceph.yml
2026-03-14 10:05:41.299514 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ceph_cleanup.yml
2026-03-14 10:05:41.299537 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ceph_help.yml
2026-03-14 10:05:41.299559 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_certmanager.yml
2026-03-14 10:05:41.299582 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_certmanager_cleanup.yml
2026-03-14 10:05:41.299604 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cifmw_cleanup.yml
2026-03-14 10:05:41.299626 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cifmw_prepare.yml
2026-03-14 10:05:41.299648 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder.yml
2026-03-14 10:05:41.299670 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_cleanup.yml
2026-03-14 10:05:41.299717 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_deploy.yml
2026-03-14 10:05:41.299741 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_deploy_cleanup.yml
2026-03-14 10:05:41.299769 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_deploy_prep.yml
2026-03-14 10:05:41.299791 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_kuttl.yml
2026-03-14 10:05:41.299813 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_kuttl_run.yml
2026-03-14 10:05:41.299834 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_prep.yml
2026-03-14 10:05:41.299856 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cleanup.yml
2026-03-14 10:05:41.299878 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc.yml
2026-03-14 10:05:41.299900 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_attach_default_interface.yml
2026-03-14 10:05:41.299922 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_attach_default_interface_cleanup.yml
2026-03-14 10:05:41.299944 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_bmo_cleanup.yml
2026-03-14 10:05:41.299967 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_bmo_setup.yml
2026-03-14 10:05:41.299989 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_cleanup.yml
2026-03-14 10:05:41.300017 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_scrub.yml
2026-03-14 10:05:41.300048 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_storage.yml
2026-03-14 10:05:41.300075 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_storage_cleanup.yml
2026-03-14 10:05:41.300101 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_storage_cleanup_with_retries.yml
2026-03-14 10:05:41.300126 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_storage_release.yml
2026-03-14 10:05:41.300150 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_storage_with_retries.yml
2026-03-14 10:05:41.300179 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_deploy_cleanup.yml
2026-03-14 10:05:41.300209 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate.yml
2026-03-14 10:05:41.300238 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_cleanup.yml
2026-03-14 10:05:41.300268 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_deploy.yml
2026-03-14 10:05:41.300293 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_deploy_cleanup.yml
2026-03-14 10:05:41.300314 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_deploy_prep.yml
2026-03-14 10:05:41.300336 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_kuttl.yml
2026-03-14 10:05:41.300355 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_kuttl_run.yml
2026-03-14 10:05:41.300378 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_prep.yml
2026-03-14 10:05:41.300395 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_dns_deploy.yml
2026-03-14 10:05:41.300413 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_dns_deploy_cleanup.yml
2026-03-14 10:05:41.300430 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_dns_deploy_prep.yml
2026-03-14 10:05:41.300451 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_download_tools.yml
2026-03-14 10:05:41.300470 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_ansible_runner.yml
2026-03-14 10:05:41.300486 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_baremetal_compute.yml
2026-03-14 10:05:41.300503 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_compute.yml
2026-03-14 10:05:41.300520 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_compute_bootc.yml
2026-03-14 10:05:41.300537 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_compute_cleanup.yml
2026-03-14 10:05:41.300554 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_compute_repos.yml
2026-03-14 10:05:41.300570 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_computes_bgp.yml
2026-03-14 10:05:41.300587 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy.yml
2026-03-14 10:05:41.300605 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_baremetal.yml
2026-03-14 10:05:41.300622 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_baremetal_prep.yml
2026-03-14 10:05:41.300639 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_cleanup.yml
2026-03-14 10:05:41.300656 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_generate_keys.yml
2026-03-14 10:05:41.300673 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_instance.yml
2026-03-14 10:05:41.300711 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_networker.yml
2026-03-14 10:05:41.300728 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_networker_cleanup.yml
2026-03-14 10:05:41.300746 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_networker_prep.yml
2026-03-14 10:05:41.300763 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_prep.yml
2026-03-14 10:05:41.300780 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_networker.yml
2026-03-14 10:05:41.300797 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_networker_cleanup.yml
2026-03-14 10:05:41.300813 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_nova_discover_hosts.yml
2026-03-14 10:05:41.300831 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_patch_ansible_runner_image.yml
2026-03-14 10:05:41.300847 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_register_dns.yml
2026-03-14 10:05:41.300864 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_wait_deploy.yml
2026-03-14 10:05:41.300881 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_wait_deploy_baremetal.yml
2026-03-14 10:05:41.300897 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance.yml
2026-03-14 10:05:41.300915 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_cleanup.yml
2026-03-14 10:05:41.300931 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_deploy.yml
2026-03-14 10:05:41.300948 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_deploy_cleanup.yml
2026-03-14 10:05:41.300971 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_deploy_prep.yml
2026-03-14 10:05:41.300996 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_kuttl.yml
2026-03-14 10:05:41.301021 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_kuttl_run.yml
2026-03-14 10:05:41.301044 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_prep.yml
2026-03-14 10:05:41.301064 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat.yml
2026-03-14 10:05:41.301082 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_cleanup.yml
2026-03-14 10:05:41.301099 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_deploy.yml
2026-03-14 10:05:41.301116 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_deploy_cleanup.yml
2026-03-14 10:05:41.301133 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_deploy_prep.yml
2026-03-14 10:05:41.301149 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_kuttl.yml
2026-03-14 10:05:41.301166 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_kuttl_crc.yml
2026-03-14 10:05:41.301182 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_kuttl_run.yml
2026-03-14 10:05:41.301199 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_prep.yml
2026-03-14 10:05:41.301216 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_help.yml
2026-03-14 10:05:41.301233 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon.yml
2026-03-14 10:05:41.301250 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_cleanup.yml
2026-03-14 10:05:41.301267 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_deploy.yml
2026-03-14 10:05:41.301283 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_deploy_cleanup.yml
2026-03-14 10:05:41.301299 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_deploy_prep.yml
2026-03-14 10:05:41.301316 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_kuttl.yml
2026-03-14 10:05:41.301333 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_kuttl_run.yml
2026-03-14 10:05:41.301349 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_prep.yml
2026-03-14 10:05:41.301443 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_infra.yml
2026-03-14 10:05:41.301469 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_infra_cleanup.yml
2026-03-14 10:05:41.301488 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_infra_kuttl.yml
2026-03-14 10:05:41.301506 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_infra_kuttl_run.yml
2026-03-14 10:05:41.301523 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_infra_prep.yml
2026-03-14 10:05:41.301541 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_input.yml
2026-03-14 10:05:41.301558 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_input_cleanup.yml
2026-03-14 10:05:41.301575 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab.yml
2026-03-14 10:05:41.301596 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_cleanup.yml
2026-03-14 10:05:41.301613 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_nat64_router.yml
2026-03-14 10:05:41.301631 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_nat64_router_cleanup.yml
2026-03-14 10:05:41.301649 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_network.yml
2026-03-14 10:05:41.301666 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_network_cleanup.yml
2026-03-14 10:05:41.301707 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_sno.yml
2026-03-14 10:05:41.301727 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_sno_cleanup.yml
2026-03-14 10:05:41.301745 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic.yml
2026-03-14 10:05:41.301762 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_cleanup.yml
2026-03-14 10:05:41.301779 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_deploy.yml
2026-03-14 10:05:41.301796 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_deploy_cleanup.yml
2026-03-14 10:05:41.301813 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_deploy_prep.yml
2026-03-14 10:05:41.301831 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_kuttl.yml
2026-03-14 10:05:41.301849 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_kuttl_crc.yml
2026-03-14 10:05:41.301866 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_kuttl_run.yml
2026-03-14 10:05:41.301883 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_prep.yml
2026-03-14 10:05:41.301900 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone.yml
2026-03-14 10:05:41.301917 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_cleanup.yml
2026-03-14 10:05:41.301935 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_deploy.yml
2026-03-14 10:05:41.301952 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_deploy_cleanup.yml
2026-03-14 10:05:41.301969 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_deploy_prep.yml
2026-03-14 10:05:41.301987 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_kuttl.yml
2026-03-14 10:05:41.302004 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_kuttl_run.yml
2026-03-14 10:05:41.302022 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_prep.yml
2026-03-14 10:05:41.302039 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_kuttl_common_cleanup.yml
2026-03-14 10:05:41.302056 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_kuttl_common_prep.yml
2026-03-14 10:05:41.302072 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_kuttl_db_cleanup.yml
2026-03-14 10:05:41.302089 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_kuttl_db_prep.yml
2026-03-14 10:05:41.302106 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_loki.yml
2026-03-14 10:05:41.302127 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_loki_cleanup.yml
2026-03-14 10:05:41.302145 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_loki_deploy.yml
2026-03-14 10:05:41.302163 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_loki_deploy_cleanup.yml
2026-03-14 10:05:41.302179 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_lvms.yml
2026-03-14 10:05:41.302197 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila.yml
2026-03-14 10:05:41.302214 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_cleanup.yml
2026-03-14 10:05:41.302231 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_deploy.yml
2026-03-14 10:05:41.302248 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_deploy_cleanup.yml
2026-03-14 10:05:41.302265 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_deploy_prep.yml
2026-03-14 10:05:41.302283 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_kuttl.yml
2026-03-14 10:05:41.302300 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_kuttl_run.yml
2026-03-14 10:05:41.302316 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_prep.yml
2026-03-14 10:05:41.302333 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb.yml
2026-03-14 10:05:41.302350 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_chainsaw.yml
2026-03-14 10:05:41.302367 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_chainsaw_run.yml
2026-03-14 10:05:41.302384 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_cleanup.yml
2026-03-14 10:05:41.302408 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_deploy.yml
2026-03-14 10:05:41.302432 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_deploy_cleanup.yml
2026-03-14 10:05:41.302456 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_deploy_prep.yml
2026-03-14 10:05:41.302474 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_kuttl.yml
2026-03-14 10:05:41.302492 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_kuttl_run.yml
2026-03-14 10:05:41.302509 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_memcached_deploy.yml
2026-03-14 10:05:41.302527 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_memcached_deploy_cleanup.yml
2026-03-14 10:05:41.302544 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_memcached_deploy_prep.yml
2026-03-14 10:05:41.302562 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_metallb.yml
2026-03-14 10:05:41.302579 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_metallb_cleanup.yml
2026-03-14 10:05:41.302596 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_metallb_config.yml
2026-03-14 10:05:41.302614 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_metallb_config_cleanup.yml
2026-03-14 10:05:41.302631 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mirror_registry.yml
2026-03-14 10:05:41.302649 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mirror_registry_ca.yml
2026-03-14 10:05:41.302671 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mirror_registry_cleanup.yml
2026-03-14 10:05:41.302710 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mirror_registry_secure.yml
2026-03-14 10:05:41.302730 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_namespace.yml
2026-03-14 10:05:41.302748 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_namespace_cleanup.yml
2026-03-14 10:05:41.302766 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netattach.yml
2026-03-14 10:05:41.302783 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netattach_cleanup.yml
2026-03-14 10:05:41.302801 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netconfig_deploy.yml
2026-03-14 10:05:41.302818 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netconfig_deploy_cleanup.yml
2026-03-14 10:05:41.302835 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netconfig_deploy_prep.yml
2026-03-14 10:05:41.302852 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netobserv.yml
2026-03-14 10:05:41.302869 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netobserv_cleanup.yml
2026-03-14 10:05:41.302886 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netobserv_deploy.yml
2026-03-14 10:05:41.302903 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netobserv_deploy_cleanup.yml
2026-03-14 10:05:41.302920 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_network_isolation_bridge.yml
2026-03-14 10:05:41.302937 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_network_isolation_bridge_cleanup.yml
2026-03-14 10:05:41.302954 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron.yml
2026-03-14 10:05:41.302971 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_cleanup.yml
2026-03-14 10:05:41.302988 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_deploy.yml
2026-03-14 10:05:41.303005 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_deploy_cleanup.yml
2026-03-14 10:05:41.303021 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_deploy_prep.yml
2026-03-14 10:05:41.303038 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_kuttl.yml
2026-03-14 10:05:41.303072 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_kuttl_run.yml
2026-03-14 10:05:41.303091 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_prep.yml
2026-03-14 10:05:41.303108 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nfs.yml
2026-03-14 10:05:41.303125 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nfs_cleanup.yml
2026-03-14 10:05:41.303143 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nmstate.yml
2026-03-14 10:05:41.303160 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nncp.yml
2026-03-14 10:05:41.303177 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nncp_cleanup.yml
2026-03-14 10:05:41.303194 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova.yml
2026-03-14 10:05:41.303216 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova_cleanup.yml
2026-03-14 10:05:41.303233 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova_deploy.yml
2026-03-14 10:05:41.303250 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova_deploy_cleanup.yml
2026-03-14 10:05:41.303267 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova_deploy_prep.yml
2026-03-14 10:05:41.303283 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova_prep.yml
2026-03-14 10:05:41.303300 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia.yml
2026-03-14 10:05:41.303316 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_cleanup.yml
2026-03-14 10:05:41.303333 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_deploy.yml
2026-03-14 10:05:41.303350 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_deploy_cleanup.yml
2026-03-14 10:05:41.303366 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_deploy_prep.yml
2026-03-14 10:05:41.303384 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_kuttl.yml
2026-03-14 10:05:41.303401 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_kuttl_run.yml
2026-03-14 10:05:41.303417 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_prep.yml
2026-03-14 10:05:41.303435 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack.yml
2026-03-14 10:05:41.303452 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_cleanup.yml
2026-03-14 10:05:41.303468 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_crds.yml
2026-03-14 10:05:41.303484 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_crds_cleanup.yml
2026-03-14 10:05:41.303501 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_deploy.yml
2026-03-14 10:05:41.303518 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_deploy_cleanup.yml
2026-03-14 10:05:41.303534 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_deploy_prep.yml
2026-03-14 10:05:41.303551 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_init.yml
2026-03-14 10:05:41.303568 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_kuttl.yml
2026-03-14 10:05:41.303585 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_kuttl_cleanup.yml
2026-03-14 10:05:41.303602 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_kuttl_prep.yml
2026-03-14 10:05:41.303620 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_kuttl_run.yml
2026-03-14 10:05:41.303637 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_patch_version.yml
2026-03-14 10:05:41.303653 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_prep.yml
2026-03-14 10:05:41.303670 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_repo.yml
2026-03-14 10:05:41.303718 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_update_run.yml
2026-03-14 10:05:41.303751 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_wait.yml
2026-03-14 10:05:41.303773 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_wait_deploy.yml
2026-03-14 10:05:41.303793 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_operator_namespace.yml
2026-03-14 10:05:41.303814 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn.yml
2026-03-14 10:05:41.303835 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_cleanup.yml
2026-03-14 10:05:41.303857 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_deploy.yml
2026-03-14 10:05:41.303878 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_deploy_cleanup.yml
2026-03-14 10:05:41.303901 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_deploy_prep.yml
2026-03-14 10:05:41.303918 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_kuttl.yml
2026-03-14 10:05:41.303934 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_kuttl_run.yml
2026-03-14 10:05:41.303958 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_prep.yml
2026-03-14 10:05:41.303977 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement.yml
2026-03-14 10:05:41.303993 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_cleanup.yml
2026-03-14 10:05:41.304010 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_deploy.yml
2026-03-14 10:05:41.304027 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_deploy_cleanup.yml
2026-03-14 10:05:41.304044 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_deploy_prep.yml
2026-03-14 10:05:41.304061 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_kuttl.yml
2026-03-14 10:05:41.304078 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_kuttl_run.yml
2026-03-14 10:05:41.304095 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_prep.yml
2026-03-14 10:05:41.304113 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq.yml
2026-03-14 10:05:41.304130 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq_cleanup.yml
2026-03-14 10:05:41.304147 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq_deploy.yml
2026-03-14 10:05:41.304164 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq_deploy_cleanup.yml
2026-03-14 10:05:41.304182 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq_deploy_prep.yml
2026-03-14 10:05:41.304199 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq_prep.yml
2026-03-14 10:05:41.304218 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_redis_deploy.yml
2026-03-14 10:05:41.304237 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_redis_deploy_cleanup.yml
2026-03-14 10:05:41.304257 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_redis_deploy_prep.yml
2026-03-14 10:05:41.304281 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook.yml
2026-03-14 10:05:41.304301 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook_cleanup.yml
2026-03-14 10:05:41.304327 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook_crc_disk.yml
2026-03-14 10:05:41.304345 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook_deploy.yml
2026-03-14 10:05:41.304362 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook_deploy_prep.yml
2026-03-14 10:05:41.304378 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook_prep.yml
2026-03-14 10:05:41.304395 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_set_slower_etcd_profile.yml
2026-03-14 10:05:41.304412 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone.yml
2026-03-14 10:05:41.304429 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone_cleanup.yml
2026-03-14 10:05:41.304446 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone_deploy.yml
2026-03-14 10:05:41.304463 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone_revert.yml
2026-03-14 10:05:41.304480 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone_snapshot.yml
2026-03-14 10:05:41.304497 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone_sync.yml
2026-03-14 10:05:41.304514 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift.yml
2026-03-14 10:05:41.304530 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_cleanup.yml
2026-03-14 10:05:41.304547 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_deploy.yml
2026-03-14 10:05:41.304564 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_deploy_cleanup.yml
2026-03-14 10:05:41.304580 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_deploy_prep.yml
2026-03-14 10:05:41.304596 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_kuttl.yml
2026-03-14 10:05:41.304612 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_kuttl_run.yml
2026-03-14 10:05:41.304629 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_prep.yml
2026-03-14 10:05:41.304645 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry.yml
2026-03-14 10:05:41.304661 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_cleanup.yml
2026-03-14 10:05:41.304699 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_deploy.yml
2026-03-14 10:05:41.304730 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_deploy_cleanup.yml
2026-03-14 10:05:41.304751 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_deploy_prep.yml
2026-03-14 10:05:41.304768 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_kuttl.yml
2026-03-14 10:05:41.304785 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_kuttl_run.yml
2026-03-14 10:05:41.304802 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_prep.yml
2026-03-14 10:05:41.304819 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_tripleo_deploy.yml
2026-03-14 10:05:41.304836 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_update_services.yml
2026-03-14 10:05:41.304857 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_update_system.yml
2026-03-14 10:05:41.304874 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_validate_marketplace.yml
2026-03-14 10:05:41.304892 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_wait.yml
2026-03-14 10:05:41.304909 | controller | cd+++++++++ ci-framework-data/artifacts/yum_repos/
2026-03-14 10:05:41.304929 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/delorean-antelope-testing.repo
2026-03-14 10:05:41.304947 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/delorean.repo
2026-03-14 10:05:41.304965 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/delorean.repo.md5
2026-03-14 10:05:41.304982 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/repo-setup-centos-appstream.repo
2026-03-14 10:05:41.304999 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/repo-setup-centos-baseos.repo
2026-03-14 10:05:41.305016 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/repo-setup-centos-highavailability.repo
2026-03-14 10:05:41.305033 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/repo-setup-centos-powertools.repo
2026-03-14 10:05:41.305050 | controller | cd+++++++++ ci-framework-data/logs/
2026-03-14 10:05:41.305067 | controller | >f+++++++++ ci-framework-data/logs/ansible.log.gz
2026-03-14 10:05:41.305084 | controller | >f+++++++++ ci-framework-data/logs/ci_script_000_copy_logs_from_crc.log
2026-03-14 10:05:41.305101 | controller | >f+++++++++ ci-framework-data/logs/ci_script_000_prepare_root_ssh.log
2026-03-14 10:05:41.305117 | controller | >f+++++++++ ci-framework-data/logs/ci_script_000_run_download.log
2026-03-14 10:05:41.305134 | controller | >f+++++++++ ci-framework-data/logs/ci_script_000_run_openstack_must_gather.log
2026-03-14 10:05:41.305151 | controller | >f+++++++++ ci-framework-data/logs/ci_script_001_run_hook_without_retry.log
2026-03-14 10:05:41.305167 | controller | >f+++++++++ ci-framework-data/logs/ci_script_002_fetch_openshift.log
2026-03-14 10:05:41.305185 | controller | >f+++++++++ ci-framework-data/logs/ci_script_003_run_openstack.log
2026-03-14 10:05:41.305201 | controller | >f+++++++++ ci-framework-data/logs/ci_script_004_run_hook_without_retry_deploy.log
2026-03-14 10:05:41.305218 | controller | >f+++++++++ ci-framework-data/logs/ci_script_005_stop_watcher.log
2026-03-14 10:05:41.305234 | controller | >f+++++++++ ci-framework-data/logs/ci_script_006_install_watcher_operator_in.log
2026-03-14 10:05:41.305251 | controller | >f+++++++++ ci-framework-data/logs/post_install_operators_kuttl_from_operator_deploy_watcher_operator.log
2026-03-14 10:05:41.305268 | controller | >f+++++++++ ci-framework-data/logs/pre_infra_download_needed_tools.log
2026-03-14 10:05:41.305285 | controller | cd+++++++++ ci-framework-data/logs/2026-03-14_09-56/
2026-03-14 10:05:41.305302 | controller | >f+++++++++ ci-framework-data/logs/2026-03-14_09-56/ansible.log.gz
2026-03-14 10:05:41.305319 | controller | cd+++++++++ ci-framework-data/logs/crc/
2026-03-14 10:05:41.305336 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/
2026-03-14 10:05:41.305353 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/
2026-03-14 10:05:41.305371 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager-operator_cert-manager-operator-controller-manager-66c8bdd694-mj4wd_6b107285-e490-48f4-a158-308c62dd492e/
2026-03-14 10:05:41.305388 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager-operator_cert-manager-operator-controller-manager-66c8bdd694-mj4wd_6b107285-e490-48f4-a158-308c62dd492e/cert-manager-operator/
2026-03-14 10:05:41.305406 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager-operator_cert-manager-operator-controller-manager-66c8bdd694-mj4wd_6b107285-e490-48f4-a158-308c62dd492e/cert-manager-operator/0.log
2026-03-14 10:05:41.305427 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-545d4d4674-d6lvd_94bce3e7-2401-4f5c-919f-6cadeb6472f4/
2026-03-14 10:05:41.305446 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-545d4d4674-d6lvd_94bce3e7-2401-4f5c-919f-6cadeb6472f4/cert-manager-controller/
2026-03-14 10:05:41.305463 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-545d4d4674-d6lvd_94bce3e7-2401-4f5c-919f-6cadeb6472f4/cert-manager-controller/0.log
2026-03-14 10:05:41.305480 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-cainjector-5545bd876-r4hhn_3520f5cf-b66e-4922-b114-2e7748a9103e/
2026-03-14 10:05:41.305497 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-cainjector-5545bd876-r4hhn_3520f5cf-b66e-4922-b114-2e7748a9103e/cert-manager-cainjector/
2026-03-14 10:05:41.305513 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-cainjector-5545bd876-r4hhn_3520f5cf-b66e-4922-b114-2e7748a9103e/cert-manager-cainjector/0.log
2026-03-14 10:05:41.305531 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-webhook-6888856db4-2rbp4_9b200cf6-c7bf-4503-9694-fb0f63d901cc/
2026-03-14 10:05:41.305547 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-webhook-6888856db4-2rbp4_9b200cf6-c7bf-4503-9694-fb0f63d901cc/cert-manager-webhook/
2026-03-14 10:05:41.305565 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-webhook-6888856db4-2rbp4_9b200cf6-c7bf-4503-9694-fb0f63d901cc/cert-manager-webhook/0.log
2026-03-14 10:05:41.305581 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-swrz2_5921a039-7ee3-49ce-aad8-cf711e43e797/
2026-03-14 10:05:41.305599 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-swrz2_5921a039-7ee3-49ce-aad8-cf711e43e797/csi-provisioner/
2026-03-14 10:05:41.305615 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-swrz2_5921a039-7ee3-49ce-aad8-cf711e43e797/csi-provisioner/0.log
2026-03-14 10:05:41.305632 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-swrz2_5921a039-7ee3-49ce-aad8-cf711e43e797/hostpath-provisioner/
2026-03-14 10:05:41.305649 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-swrz2_5921a039-7ee3-49ce-aad8-cf711e43e797/hostpath-provisioner/0.log
2026-03-14 10:05:41.305666 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-swrz2_5921a039-7ee3-49ce-aad8-cf711e43e797/liveness-probe/
2026-03-14 10:05:41.305706 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-swrz2_5921a039-7ee3-49ce-aad8-cf711e43e797/liveness-probe/0.log
2026-03-14 10:05:41.305732 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-swrz2_5921a039-7ee3-49ce-aad8-cf711e43e797/node-driver-registrar/
2026-03-14 10:05:41.305754 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-swrz2_5921a039-7ee3-49ce-aad8-cf711e43e797/node-driver-registrar/0.log
2026-03-14 10:05:41.305773 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_controller-7bb4cc7c98-8v9j7_64b1131f-66f6-4bdf-b60e-f5d8f8680eeb/
2026-03-14 10:05:41.305789 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_controller-7bb4cc7c98-8v9j7_64b1131f-66f6-4bdf-b60e-f5d8f8680eeb/controller/
2026-03-14 10:05:41.305812 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_controller-7bb4cc7c98-8v9j7_64b1131f-66f6-4bdf-b60e-f5d8f8680eeb/controller/0.log.gz
2026-03-14 10:05:41.305830 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_controller-7bb4cc7c98-8v9j7_64b1131f-66f6-4bdf-b60e-f5d8f8680eeb/kube-rbac-proxy/
2026-03-14 10:05:41.305847 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_controller-7bb4cc7c98-8v9j7_64b1131f-66f6-4bdf-b60e-f5d8f8680eeb/kube-rbac-proxy/0.log
2026-03-14 10:05:41.305864 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-2llz5_63a81c2e-7222-4852-b557-f6a24194d46f/
2026-03-14 10:05:41.305881 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-2llz5_63a81c2e-7222-4852-b557-f6a24194d46f/fd008d54b5c9ce448eafb29acced479998928b64de38b315393ecab506aee1dc.log
2026-03-14 10:05:41.305897 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-2llz5_63a81c2e-7222-4852-b557-f6a24194d46f/controller/
2026-03-14 10:05:41.305914 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-2llz5_63a81c2e-7222-4852-b557-f6a24194d46f/controller/0.log
2026-03-14 10:05:41.305930 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-2llz5_63a81c2e-7222-4852-b557-f6a24194d46f/cp-frr-files/
2026-03-14 10:05:41.305948 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-2llz5_63a81c2e-7222-4852-b557-f6a24194d46f/cp-frr-files/0.log
2026-03-14 10:05:41.305965 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-2llz5_63a81c2e-7222-4852-b557-f6a24194d46f/cp-metrics/
2026-03-14 10:05:41.305982 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-2llz5_63a81c2e-7222-4852-b557-f6a24194d46f/cp-metrics/0.log
2026-03-14 10:05:41.305998 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-2llz5_63a81c2e-7222-4852-b557-f6a24194d46f/cp-reloader/
2026-03-14 10:05:41.306015 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-2llz5_63a81c2e-7222-4852-b557-f6a24194d46f/cp-reloader/0.log
2026-03-14 10:05:41.306031 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-2llz5_63a81c2e-7222-4852-b557-f6a24194d46f/frr-metrics/
2026-03-14 10:05:41.306047 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-2llz5_63a81c2e-7222-4852-b557-f6a24194d46f/frr-metrics/0.log
2026-03-14 10:05:41.306064 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-2llz5_63a81c2e-7222-4852-b557-f6a24194d46f/frr/
2026-03-14 10:05:41.306081 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-2llz5_63a81c2e-7222-4852-b557-f6a24194d46f/frr/0.log.gz
2026-03-14 10:05:41.306098 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-2llz5_63a81c2e-7222-4852-b557-f6a24194d46f/kube-rbac-proxy-frr/
2026-03-14 10:05:41.306115 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-2llz5_63a81c2e-7222-4852-b557-f6a24194d46f/kube-rbac-proxy-frr/0.log
2026-03-14 10:05:41.306132 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-2llz5_63a81c2e-7222-4852-b557-f6a24194d46f/kube-rbac-proxy/
2026-03-14 10:05:41.306149 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-2llz5_63a81c2e-7222-4852-b557-f6a24194d46f/kube-rbac-proxy/0.log
2026-03-14 10:05:41.306170 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-2llz5_63a81c2e-7222-4852-b557-f6a24194d46f/reloader/
2026-03-14 10:05:41.306187 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-2llz5_63a81c2e-7222-4852-b557-f6a24194d46f/reloader/0.log
2026-03-14 10:05:41.306204 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-webhook-server-bcc4b6f68-nbb5f_2022ba85-6e88-485c-ba26-7f16fc03c257/
2026-03-14 10:05:41.306220 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-webhook-server-bcc4b6f68-nbb5f_2022ba85-6e88-485c-ba26-7f16fc03c257/frr-k8s-webhook-server/
2026-03-14 10:05:41.306237 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-webhook-server-bcc4b6f68-nbb5f_2022ba85-6e88-485c-ba26-7f16fc03c257/frr-k8s-webhook-server/0.log
2026-03-14 10:05:41.306254 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-controller-manager-6c5855cb99-gcpx9_1501c094-2079-46b0-8096-b4a09ee21e8b/
2026-03-14 10:05:41.306270 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-controller-manager-6c5855cb99-gcpx9_1501c094-2079-46b0-8096-b4a09ee21e8b/manager/
2026-03-14 10:05:41.306286 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-controller-manager-6c5855cb99-gcpx9_1501c094-2079-46b0-8096-b4a09ee21e8b/manager/0.log
2026-03-14 10:05:41.306303 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-webhook-server-76b978c99b-flxj6_ebaf629f-8937-4e21-bedb-0d5992f22edc/
2026-03-14 10:05:41.306320 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-webhook-server-76b978c99b-flxj6_ebaf629f-8937-4e21-bedb-0d5992f22edc/webhook-server/
2026-03-14 10:05:41.306337 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-webhook-server-76b978c99b-flxj6_ebaf629f-8937-4e21-bedb-0d5992f22edc/webhook-server/0.log
2026-03-14 10:05:41.306354 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_speaker-bv84h_d3444025-ba10-41ec-94cd-0ce213c73311/
2026-03-14 10:05:41.306370 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_speaker-bv84h_d3444025-ba10-41ec-94cd-0ce213c73311/kube-rbac-proxy/
2026-03-14 10:05:41.306387 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_speaker-bv84h_d3444025-ba10-41ec-94cd-0ce213c73311/kube-rbac-proxy/0.log
2026-03-14 10:05:41.306404 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_speaker-bv84h_d3444025-ba10-41ec-94cd-0ce213c73311/speaker/
2026-03-14 10:05:41.306421 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_speaker-bv84h_d3444025-ba10-41ec-94cd-0ce213c73311/speaker/0.log.gz
2026-03-14 10:05:41.306437 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver-operator_openshift-apiserver-operator-796bbdcf4f-xjk7w_e6fc3d87-f84d-4782-8cab-adf072d642b4/
2026-03-14 10:05:41.306454 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver-operator_openshift-apiserver-operator-796bbdcf4f-xjk7w_e6fc3d87-f84d-4782-8cab-adf072d642b4/openshift-apiserver-operator/
2026-03-14 10:05:41.306472 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver-operator_openshift-apiserver-operator-796bbdcf4f-xjk7w_e6fc3d87-f84d-4782-8cab-adf072d642b4/openshift-apiserver-operator/0.log
2026-03-14 10:05:41.306489 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-n64hl_ecf31fa9-e43c-4d64-95bd-4a52938b67f6/
2026-03-14 10:05:41.306511 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-n64hl_ecf31fa9-e43c-4d64-95bd-4a52938b67f6/fix-audit-permissions/
2026-03-14 10:05:41.306529 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-n64hl_ecf31fa9-e43c-4d64-95bd-4a52938b67f6/fix-audit-permissions/0.log
2026-03-14 10:05:41.306546 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-n64hl_ecf31fa9-e43c-4d64-95bd-4a52938b67f6/openshift-apiserver-check-endpoints/
2026-03-14 10:05:41.306563 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-n64hl_ecf31fa9-e43c-4d64-95bd-4a52938b67f6/openshift-apiserver-check-endpoints/0.log
2026-03-14 10:05:41.306580 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-n64hl_ecf31fa9-e43c-4d64-95bd-4a52938b67f6/openshift-apiserver/
2026-03-14 10:05:41.306597 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-n64hl_ecf31fa9-e43c-4d64-95bd-4a52938b67f6/openshift-apiserver/0.log
2026-03-14 10:05:41.306614 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication-operator_authentication-operator-69f744f599-nxkww_d978f007-fe7a-4956-afc5-c83101961ee8/
2026-03-14 10:05:41.306631 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication-operator_authentication-operator-69f744f599-nxkww_d978f007-fe7a-4956-afc5-c83101961ee8/authentication-operator/
2026-03-14 10:05:41.306648 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication-operator_authentication-operator-69f744f599-nxkww_d978f007-fe7a-4956-afc5-c83101961ee8/authentication-operator/0.log
2026-03-14 10:05:41.306665 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication_oauth-openshift-5f94f6fdf4-n49wj_44986e29-0f22-4f42-a902-2f7772310d2b/
2026-03-14 10:05:41.306703 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication_oauth-openshift-5f94f6fdf4-n49wj_44986e29-0f22-4f42-a902-2f7772310d2b/oauth-openshift/
2026-03-14 10:05:41.306722 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication_oauth-openshift-5f94f6fdf4-n49wj_44986e29-0f22-4f42-a902-2f7772310d2b/oauth-openshift/0.log
2026-03-14 10:05:41.306740 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-machine-approver_machine-approver-56656f9798-fkgxr_26b0b618-116b-4aa2-8e84-3e024a2c46fd/
2026-03-14 10:05:41.306756 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-machine-approver_machine-approver-56656f9798-fkgxr_26b0b618-116b-4aa2-8e84-3e024a2c46fd/kube-rbac-proxy/
2026-03-14 10:05:41.306774 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-machine-approver_machine-approver-56656f9798-fkgxr_26b0b618-116b-4aa2-8e84-3e024a2c46fd/kube-rbac-proxy/0.log
2026-03-14 10:05:41.306791 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-machine-approver_machine-approver-56656f9798-fkgxr_26b0b618-116b-4aa2-8e84-3e024a2c46fd/machine-approver-controller/
2026-03-14 10:05:41.306808 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-machine-approver_machine-approver-56656f9798-fkgxr_26b0b618-116b-4aa2-8e84-3e024a2c46fd/machine-approver-controller/0.log
2026-03-14 10:05:41.306825 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-vkqb9_47b9849c-fc22-472c-a1f6-bd0e7a962a75/
2026-03-14 10:05:41.306846 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-vkqb9_47b9849c-fc22-472c-a1f6-bd0e7a962a75/a5f6d41788f951126eeb25279ffe513e191d7fd95c6f756aeac42415c75a77ce.log
2026-03-14 10:05:41.306865 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-vkqb9_47b9849c-fc22-472c-a1f6-bd0e7a962a75/cluster-samples-operator-watch/
2026-03-14 10:05:41.306882 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-vkqb9_47b9849c-fc22-472c-a1f6-bd0e7a962a75/cluster-samples-operator-watch/0.log
2026-03-14 10:05:41.306900 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-vkqb9_47b9849c-fc22-472c-a1f6-bd0e7a962a75/cluster-samples-operator/
2026-03-14 10:05:41.306917 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-vkqb9_47b9849c-fc22-472c-a1f6-bd0e7a962a75/cluster-samples-operator/0.log
2026-03-14 10:05:41.306933 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-version_cluster-version-operator-5c965bbfc6-h6gk2_d155422d-29df-4707-8b98-524a7dee63e8/
2026-03-14 10:05:41.306950 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-version_cluster-version-operator-5c965bbfc6-h6gk2_d155422d-29df-4707-8b98-524a7dee63e8/cluster-version-operator/
2026-03-14 10:05:41.306967 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-version_cluster-version-operator-5c965bbfc6-h6gk2_d155422d-29df-4707-8b98-524a7dee63e8/cluster-version-operator/0.log.gz
2026-03-14 10:05:41.306984 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-config-operator_openshift-config-operator-7777fb866f-ph27v_20cc2773-132c-4a74-9691-748a73111124/
2026-03-14 10:05:41.307001 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-config-operator_openshift-config-operator-7777fb866f-ph27v_20cc2773-132c-4a74-9691-748a73111124/openshift-api/
2026-03-14 10:05:41.307019 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-config-operator_openshift-config-operator-7777fb866f-ph27v_20cc2773-132c-4a74-9691-748a73111124/openshift-api/0.log
2026-03-14 10:05:41.307036 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-config-operator_openshift-config-operator-7777fb866f-ph27v_20cc2773-132c-4a74-9691-748a73111124/openshift-config-operator/
2026-03-14 10:05:41.307067 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-config-operator_openshift-config-operator-7777fb866f-ph27v_20cc2773-132c-4a74-9691-748a73111124/openshift-config-operator/0.log
2026-03-14 10:05:41.307086 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console-operator_console-operator-58897d9998-ghbdz_a46e0a56-02cc-43c0-b7ab-1c792b04fa2b/
2026-03-14 10:05:41.307102 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console-operator_console-operator-58897d9998-ghbdz_a46e0a56-02cc-43c0-b7ab-1c792b04fa2b/console-operator/
2026-03-14 10:05:41.307119 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console-operator_console-operator-58897d9998-ghbdz_a46e0a56-02cc-43c0-b7ab-1c792b04fa2b/console-operator/0.log
2026-03-14 10:05:41.307136 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_console-7cff745649-s9lhf_c3ea7d60-60f0-415b-ad47-e0f383fb4fca/
2026-03-14 10:05:41.307153 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_console-7cff745649-s9lhf_c3ea7d60-60f0-415b-ad47-e0f383fb4fca/console/
2026-03-14 10:05:41.307176 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_console-7cff745649-s9lhf_c3ea7d60-60f0-415b-ad47-e0f383fb4fca/console/0.log
2026-03-14 10:05:41.307193 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_downloads-7954f5f757-tr9zk_570cf110-147b-482b-afe9-2b0ca133b6dc/
2026-03-14 10:05:41.307210 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_downloads-7954f5f757-tr9zk_570cf110-147b-482b-afe9-2b0ca133b6dc/download-server/
2026-03-14 10:05:41.307228 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_downloads-7954f5f757-tr9zk_570cf110-147b-482b-afe9-2b0ca133b6dc/download-server/0.log
2026-03-14 10:05:41.307246 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager-operator_openshift-controller-manager-operator-756b6f6bc6-5hr2h_30f8e198-f784-4504-8fa3-09b59daa7b4d/
2026-03-14 10:05:41.307262 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager-operator_openshift-controller-manager-operator-756b6f6bc6-5hr2h_30f8e198-f784-4504-8fa3-09b59daa7b4d/openshift-controller-manager-operator/
2026-03-14 10:05:41.307279 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager-operator_openshift-controller-manager-operator-756b6f6bc6-5hr2h_30f8e198-f784-4504-8fa3-09b59daa7b4d/openshift-controller-manager-operator/0.log
2026-03-14 10:05:41.307296 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager_controller-manager-6fbf776b56-8ntpp_ab867447-e512-4976-b8fe-c7d23ace44cf/
2026-03-14 10:05:41.307313 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager_controller-manager-6fbf776b56-8ntpp_ab867447-e512-4976-b8fe-c7d23ace44cf/controller-manager/
2026-03-14 10:05:41.307330 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager_controller-manager-6fbf776b56-8ntpp_ab867447-e512-4976-b8fe-c7d23ace44cf/controller-manager/0.log
2026-03-14 10:05:41.307347 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns-operator_dns-operator-744455d44c-zkgfh_4aba2d14-caf3-485c-8ee7-725ed508d1cf/
2026-03-14 10:05:41.307364 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns-operator_dns-operator-744455d44c-zkgfh_4aba2d14-caf3-485c-8ee7-725ed508d1cf/dns-operator/
2026-03-14 10:05:41.307381 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns-operator_dns-operator-744455d44c-zkgfh_4aba2d14-caf3-485c-8ee7-725ed508d1cf/dns-operator/0.log
2026-03-14 10:05:41.307398 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns-operator_dns-operator-744455d44c-zkgfh_4aba2d14-caf3-485c-8ee7-725ed508d1cf/kube-rbac-proxy/
2026-03-14 10:05:41.307414 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns-operator_dns-operator-744455d44c-zkgfh_4aba2d14-caf3-485c-8ee7-725ed508d1cf/kube-rbac-proxy/0.log
2026-03-14 10:05:41.307431 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_dns-default-mr75n_6cd2a062-18f8-44d1-918e-02142c6a2c16/
2026-03-14 10:05:41.307448 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_dns-default-mr75n_6cd2a062-18f8-44d1-918e-02142c6a2c16/dns/
2026-03-14 10:05:41.307465 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_dns-default-mr75n_6cd2a062-18f8-44d1-918e-02142c6a2c16/dns/0.log
2026-03-14 10:05:41.307481 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_dns-default-mr75n_6cd2a062-18f8-44d1-918e-02142c6a2c16/kube-rbac-proxy/
2026-03-14 10:05:41.307501 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_dns-default-mr75n_6cd2a062-18f8-44d1-918e-02142c6a2c16/kube-rbac-proxy/0.log
2026-03-14 10:05:41.307518 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_node-resolver-2sxrf_3762367a-eb3b-4992-9f00-5dd12b8ff0b1/
2026-03-14 10:05:41.307535 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_node-resolver-2sxrf_3762367a-eb3b-4992-9f00-5dd12b8ff0b1/dns-node-resolver/
2026-03-14 10:05:41.307552 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_node-resolver-2sxrf_3762367a-eb3b-4992-9f00-5dd12b8ff0b1/dns-node-resolver/0.log
2026-03-14 10:05:41.307569 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd-operator_etcd-operator-b45778765-f2fl7_be6a1160-25cd-4a04-ad30-f1c40c8d7e05/
2026-03-14 10:05:41.307586 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd-operator_etcd-operator-b45778765-f2fl7_be6a1160-25cd-4a04-ad30-f1c40c8d7e05/etcd-operator/
2026-03-14 10:05:41.307610 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd-operator_etcd-operator-b45778765-f2fl7_be6a1160-25cd-4a04-ad30-f1c40c8d7e05/etcd-operator/0.log
2026-03-14 10:05:41.307632 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/
2026-03-14 10:05:41.307654 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-ensure-env-vars/
2026-03-14 10:05:41.307692 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-ensure-env-vars/0.log
2026-03-14 10:05:41.307715 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-metrics/
2026-03-14 10:05:41.307737 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-metrics/0.log
2026-03-14 10:05:41.307760 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-readyz/
2026-03-14 10:05:41.307781 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-readyz/0.log
2026-03-14 10:05:41.307803 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-resources-copy/
2026-03-14 10:05:41.307825 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-resources-copy/0.log
2026-03-14 10:05:41.307848 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-rev/
2026-03-14 10:05:41.307869 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-rev/0.log
2026-03-14 10:05:41.307891 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd/
2026-03-14 10:05:41.307914 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd/0.log
2026-03-14 10:05:41.307936 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcdctl/
2026-03-14 10:05:41.307959 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcdctl/0.log
2026-03-14 10:05:41.307981 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/setup/
2026-03-14 10:05:41.308072 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/setup/0.log
2026-03-14 10:05:41.308098 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_cluster-image-registry-operator-dc59b4c8b-tnfbk_e21032b1-6e5a-480c-92df-d60d83844fc2/
2026-03-14 10:05:41.308117 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_cluster-image-registry-operator-dc59b4c8b-tnfbk_e21032b1-6e5a-480c-92df-d60d83844fc2/cluster-image-registry-operator/
2026-03-14 10:05:41.308135 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_cluster-image-registry-operator-dc59b4c8b-tnfbk_e21032b1-6e5a-480c-92df-d60d83844fc2/cluster-image-registry-operator/0.log
2026-03-14 10:05:41.308153 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_image-registry-66df7c8f76-8qk6m_aef4eb4a-915e-4768-ae8b-8c166b4fa8f2/
2026-03-14 10:05:41.308171 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_image-registry-66df7c8f76-8qk6m_aef4eb4a-915e-4768-ae8b-8c166b4fa8f2/registry/
2026-03-14 10:05:41.308188 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_image-registry-66df7c8f76-8qk6m_aef4eb4a-915e-4768-ae8b-8c166b4fa8f2/registry/0.log
2026-03-14 10:05:41.308206 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_node-ca-xdc5m_d34f055d-2583-40a9-948d-3abc010b3996/
2026-03-14 10:05:41.308223 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_node-ca-xdc5m_d34f055d-2583-40a9-948d-3abc010b3996/node-ca/
2026-03-14 10:05:41.308240 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_node-ca-xdc5m_d34f055d-2583-40a9-948d-3abc010b3996/node-ca/0.log
2026-03-14 10:05:41.308257 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558036-9tltz_367c08b3-ceb2-4787-841b-c8bb6d3ce971/
2026-03-14 10:05:41.308275 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558036-9tltz_367c08b3-ceb2-4787-841b-c8bb6d3ce971/oc/
2026-03-14 10:05:41.308292 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558036-9tltz_367c08b3-ceb2-4787-841b-c8bb6d3ce971/oc/0.log
2026-03-14 10:05:41.308309 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558038-thp2w_c6fea642-5dda-4baa-8b0e-9fe644a58a2f/
2026-03-14 10:05:41.308327 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558038-thp2w_c6fea642-5dda-4baa-8b0e-9fe644a58a2f/oc/
2026-03-14 10:05:41.308344 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558038-thp2w_c6fea642-5dda-4baa-8b0e-9fe644a58a2f/oc/0.log
2026-03-14 10:05:41.308360 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558040-vkx4b_0759f8a2-a59e-4675-b1ac-75ba086b6364/
2026-03-14 10:05:41.308378 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558040-vkx4b_0759f8a2-a59e-4675-b1ac-75ba086b6364/oc/
2026-03-14 10:05:41.308396 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558040-vkx4b_0759f8a2-a59e-4675-b1ac-75ba086b6364/oc/0.log
2026-03-14 10:05:41.308412 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558042-gbkvh_b20e688b-1831-41ce-8b34-e3af8a9fd5c5/
2026-03-14 10:05:41.308434 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558042-gbkvh_b20e688b-1831-41ce-8b34-e3af8a9fd5c5/oc/
2026-03-14 10:05:41.308452 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558042-gbkvh_b20e688b-1831-41ce-8b34-e3af8a9fd5c5/oc/0.log
2026-03-14 10:05:41.308468 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-canary_ingress-canary-46z5v_e8eefdff-74f7-4bb8-b39f-753a1375a403/
2026-03-14 10:05:41.308485 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-canary_ingress-canary-46z5v_e8eefdff-74f7-4bb8-b39f-753a1375a403/serve-healthcheck-canary/
2026-03-14 10:05:41.308502 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-canary_ingress-canary-46z5v_e8eefdff-74f7-4bb8-b39f-753a1375a403/serve-healthcheck-canary/0.log
2026-03-14 10:05:41.308520 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-operator_ingress-operator-5b745b69d9-jp4wl_f1c2692c-98ff-4bd0-8c81-8249f124b464/
2026-03-14 10:05:41.308537 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-operator_ingress-operator-5b745b69d9-jp4wl_f1c2692c-98ff-4bd0-8c81-8249f124b464/ingress-operator/
2026-03-14 10:05:41.308554 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-operator_ingress-operator-5b745b69d9-jp4wl_f1c2692c-98ff-4bd0-8c81-8249f124b464/ingress-operator/0.log
2026-03-14 10:05:41.308571 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-operator_ingress-operator-5b745b69d9-jp4wl_f1c2692c-98ff-4bd0-8c81-8249f124b464/kube-rbac-proxy/
2026-03-14 10:05:41.308588 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-operator_ingress-operator-5b745b69d9-jp4wl_f1c2692c-98ff-4bd0-8c81-8249f124b464/kube-rbac-proxy/0.log
2026-03-14 10:05:41.308606 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress_router-default-5444994796-pwxsw_fb7125f9-9cca-41e8-9cb0-af829e1b7287/
2026-03-14 10:05:41.308622 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress_router-default-5444994796-pwxsw_fb7125f9-9cca-41e8-9cb0-af829e1b7287/router/
2026-03-14 10:05:41.308639 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress_router-default-5444994796-pwxsw_fb7125f9-9cca-41e8-9cb0-af829e1b7287/router/0.log
2026-03-14 10:05:41.308656 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver-operator_kube-apiserver-operator-766d6c64bb-4zfx7_7dd09639-861b-4f3d-b001-e28c8650cad7/
2026-03-14 10:05:41.308673 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver-operator_kube-apiserver-operator-766d6c64bb-4zfx7_7dd09639-861b-4f3d-b001-e28c8650cad7/kube-apiserver-operator/
2026-03-14 10:05:41.308712 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver-operator_kube-apiserver-operator-766d6c64bb-4zfx7_7dd09639-861b-4f3d-b001-e28c8650cad7/kube-apiserver-operator/0.log
2026-03-14 10:05:41.308730 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_installer-9-crc_d6032032-cdfd-41cd-8783-04efb653d09f/
2026-03-14 10:05:41.308747 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_installer-9-crc_d6032032-cdfd-41cd-8783-04efb653d09f/installer/
2026-03-14 10:05:41.308765 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_installer-9-crc_d6032032-cdfd-41cd-8783-04efb653d09f/installer/0.log
2026-03-14 10:05:41.308782 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/
2026-03-14 10:05:41.308804 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-cert-regeneration-controller/
2026-03-14 10:05:41.308822 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-cert-regeneration-controller/0.log
2026-03-14 10:05:41.308839 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-cert-syncer/
2026-03-14 10:05:41.308857 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-cert-syncer/0.log
2026-03-14 10:05:41.308874 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-check-endpoints/
2026-03-14 10:05:41.308891 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-check-endpoints/0.log
2026-03-14 10:05:41.308907 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-insecure-readyz/
2026-03-14 10:05:41.308924 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-insecure-readyz/0.log
2026-03-14 10:05:41.308941 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver/
2026-03-14 10:05:41.308957 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver/0.log
2026-03-14 10:05:41.308975 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/setup/
2026-03-14 10:05:41.308992 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/setup/0.log
2026-03-14 10:05:41.309008 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-8-crc_ad8ab5cf-bf5d-405b-a2ff-d2d394731263/
2026-03-14 10:05:41.309025 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-8-crc_ad8ab5cf-bf5d-405b-a2ff-d2d394731263/pruner/
2026-03-14 10:05:41.309043 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-8-crc_ad8ab5cf-bf5d-405b-a2ff-d2d394731263/pruner/0.log
2026-03-14 10:05:41.309060 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-9-crc_684ccdc3-1bcb-4227-8679-b9d31a21198c/
2026-03-14 10:05:41.309077 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-9-crc_684ccdc3-1bcb-4227-8679-b9d31a21198c/pruner/
2026-03-14 10:05:41.309094 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-9-crc_684ccdc3-1bcb-4227-8679-b9d31a21198c/pruner/0.log
2026-03-14 10:05:41.309111 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager-operator_kube-controller-manager-operator-78b949d7b-j5cd7_15553d9e-d0a1-495b-8c4f-35f3d026fb33/
2026-03-14 10:05:41.309132 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager-operator_kube-controller-manager-operator-78b949d7b-j5cd7_15553d9e-d0a1-495b-8c4f-35f3d026fb33/kube-controller-manager-operator/
2026-03-14 10:05:41.309150 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager-operator_kube-controller-manager-operator-78b949d7b-j5cd7_15553d9e-d0a1-495b-8c4f-35f3d026fb33/kube-controller-manager-operator/0.log
2026-03-14 10:05:41.309167 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/
2026-03-14 10:05:41.309184 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/cluster-policy-controller/
2026-03-14 10:05:41.309201 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/cluster-policy-controller/1.log
2026-03-14 10:05:41.309218 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/cluster-policy-controller/2.log
2026-03-14 10:05:41.309235 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager-cert-syncer/
2026-03-14 10:05:41.309253 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager-cert-syncer/0.log
2026-03-14 10:05:41.309270 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager-recovery-controller/
2026-03-14 10:05:41.309287 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager-recovery-controller/0.log
2026-03-14 10:05:41.309304 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager/
2026-03-14 10:05:41.309321 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager/0.log
2026-03-14 10:05:41.309338 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager/1.log
2026-03-14 10:05:41.309354 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_revision-pruner-9-crc_c6fe3ef4-be53-4048-ad60-5bdf196cf10e/
2026-03-14 10:05:41.309371 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_revision-pruner-9-crc_c6fe3ef4-be53-4048-ad60-5bdf196cf10e/pruner/
2026-03-14 10:05:41.309388 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_revision-pruner-9-crc_c6fe3ef4-be53-4048-ad60-5bdf196cf10e/pruner/0.log
2026-03-14 10:05:41.309404 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler-operator_openshift-kube-scheduler-operator-5fdd9b5758-5v9l7_6e938b24-b7e8-4f6e-9c4a-2a93410278a4/
2026-03-14 10:05:41.309421 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler-operator_openshift-kube-scheduler-operator-5fdd9b5758-5v9l7_6e938b24-b7e8-4f6e-9c4a-2a93410278a4/kube-scheduler-operator-container/
2026-03-14 10:05:41.309443 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler-operator_openshift-kube-scheduler-operator-5fdd9b5758-5v9l7_6e938b24-b7e8-4f6e-9c4a-2a93410278a4/kube-scheduler-operator-container/0.log
2026-03-14 10:05:41.309460 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/
2026-03-14 10:05:41.309478 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler-cert-syncer/
2026-03-14 10:05:41.309496 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler-cert-syncer/0.log
2026-03-14 10:05:41.309513 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler-recovery-controller/
2026-03-14 10:05:41.309530 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler-recovery-controller/0.log
2026-03-14 10:05:41.309548 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler/
2026-03-14 10:05:41.309565 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler/0.log
2026-03-14 10:05:41.309582 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/wait-for-host-port/
2026-03-14 10:05:41.309599 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/wait-for-host-port/0.log
2026-03-14 10:05:41.309617 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-b67b599dd-kd9zf_6e221699-471c-4cec-9aa8-b680923ac8b9/
2026-03-14 10:05:41.309634 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-b67b599dd-kd9zf_6e221699-471c-4cec-9aa8-b680923ac8b9/kube-storage-version-migrator-operator/
2026-03-14 10:05:41.309652 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-b67b599dd-kd9zf_6e221699-471c-4cec-9aa8-b680923ac8b9/kube-storage-version-migrator-operator/0.log
2026-03-14 10:05:41.309669 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator_migrator-59844c95c7-qxscp_8d64bc52-9c28-4efb-8f3f-aeeb8a738fc6/
2026-03-14 10:05:41.309706 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator_migrator-59844c95c7-qxscp_8d64bc52-9c28-4efb-8f3f-aeeb8a738fc6/graceful-termination/
2026-03-14 10:05:41.309725 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator_migrator-59844c95c7-qxscp_8d64bc52-9c28-4efb-8f3f-aeeb8a738fc6/graceful-termination/0.log
2026-03-14 10:05:41.309743 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator_migrator-59844c95c7-qxscp_8d64bc52-9c28-4efb-8f3f-aeeb8a738fc6/migrator/
2026-03-14 10:05:41.309760 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator_migrator-59844c95c7-qxscp_8d64bc52-9c28-4efb-8f3f-aeeb8a738fc6/migrator/0.log
2026-03-14 10:05:41.309782 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_control-plane-machine-set-operator-78cbb6b69f-cxr6c_82f06b03-5a8e-4ef6-aac7-3c0b9326183e/
2026-03-14 10:05:41.309800 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_control-plane-machine-set-operator-78cbb6b69f-cxr6c_82f06b03-5a8e-4ef6-aac7-3c0b9326183e/control-plane-machine-set-operator/
2026-03-14 10:05:41.309817 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_control-plane-machine-set-operator-78cbb6b69f-cxr6c_82f06b03-5a8e-4ef6-aac7-3c0b9326183e/control-plane-machine-set-operator/0.log
2026-03-14 10:05:41.309835 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_machine-api-operator-5694c8668f-5kqvs_ce314766-eb86-4b0a-909b-2ae24850a4af/
2026-03-14 10:05:41.309853 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_machine-api-operator-5694c8668f-5kqvs_ce314766-eb86-4b0a-909b-2ae24850a4af/kube-rbac-proxy/
2026-03-14 10:05:41.309871 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_machine-api-operator-5694c8668f-5kqvs_ce314766-eb86-4b0a-909b-2ae24850a4af/kube-rbac-proxy/0.log
2026-03-14 10:05:41.309888 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_machine-api-operator-5694c8668f-5kqvs_ce314766-eb86-4b0a-909b-2ae24850a4af/machine-api-operator/
2026-03-14 10:05:41.309906 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_machine-api-operator-5694c8668f-5kqvs_ce314766-eb86-4b0a-909b-2ae24850a4af/machine-api-operator/0.log
2026-03-14 10:05:41.309923 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-crc_d1b160f5dda77d281dd8e69ec8d817f9/
2026-03-14 10:05:41.309941 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-crc_d1b160f5dda77d281dd8e69ec8d817f9/kube-rbac-proxy-crio/
2026-03-14 10:05:41.309958 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-crc_d1b160f5dda77d281dd8e69ec8d817f9/kube-rbac-proxy-crio/0.log
2026-03-14 10:05:41.309975 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-crc_d1b160f5dda77d281dd8e69ec8d817f9/setup/
2026-03-14 10:05:41.309992 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-crc_d1b160f5dda77d281dd8e69ec8d817f9/setup/0.log
2026-03-14 10:05:41.310009 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-controller-84d6567774-xvzgq_fb768ff1-000e-4bef-9c37-fcfb4fdd7e76/
2026-03-14 10:05:41.310027 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-controller-84d6567774-xvzgq_fb768ff1-000e-4bef-9c37-fcfb4fdd7e76/kube-rbac-proxy/
2026-03-14 10:05:41.310044 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-controller-84d6567774-xvzgq_fb768ff1-000e-4bef-9c37-fcfb4fdd7e76/kube-rbac-proxy/0.log
2026-03-14 10:05:41.310061 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-controller-84d6567774-xvzgq_fb768ff1-000e-4bef-9c37-fcfb4fdd7e76/machine-config-controller/
2026-03-14 10:05:41.310078 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-controller-84d6567774-xvzgq_fb768ff1-000e-4bef-9c37-fcfb4fdd7e76/machine-config-controller/0.log
2026-03-14 10:05:41.310111 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-gwd22_8f55462c-aaf9-41fa-bae6-2f7fa118841b/
2026-03-14 10:05:41.310129 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-gwd22_8f55462c-aaf9-41fa-bae6-2f7fa118841b/kube-rbac-proxy/
2026-03-14 10:05:41.310147 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-gwd22_8f55462c-aaf9-41fa-bae6-2f7fa118841b/kube-rbac-proxy/0.log
2026-03-14 10:05:41.310164 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-gwd22_8f55462c-aaf9-41fa-bae6-2f7fa118841b/machine-config-daemon/
2026-03-14 10:05:41.310182 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-gwd22_8f55462c-aaf9-41fa-bae6-2f7fa118841b/machine-config-daemon/11.log
2026-03-14 10:05:41.310199 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-operator-74547568cd-7xgrx_66202f9d-911e-47f7-b2d2-6326dd9c7283/
2026-03-14 10:05:41.310216 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-operator-74547568cd-7xgrx_66202f9d-911e-47f7-b2d2-6326dd9c7283/kube-rbac-proxy/
2026-03-14 10:05:41.310235 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-operator-74547568cd-7xgrx_66202f9d-911e-47f7-b2d2-6326dd9c7283/kube-rbac-proxy/0.log
2026-03-14 10:05:41.310255 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-operator-74547568cd-7xgrx_66202f9d-911e-47f7-b2d2-6326dd9c7283/machine-config-operator/
2026-03-14 10:05:41.310273 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-operator-74547568cd-7xgrx_66202f9d-911e-47f7-b2d2-6326dd9c7283/machine-config-operator/0.log
2026-03-14 10:05:41.310290 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-server-xv469_d11b2fa2-c2dc-4fe6-b195-d5da5e91f2b5/
2026-03-14 10:05:41.310308 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-server-xv469_d11b2fa2-c2dc-4fe6-b195-d5da5e91f2b5/machine-config-server/
2026-03-14 10:05:41.310325 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-server-xv469_d11b2fa2-c2dc-4fe6-b195-d5da5e91f2b5/machine-config-server/0.log
2026-03-14 10:05:41.310342 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874cs2f2_e952e216-1adb-456a-9f7e-61d2dc2ddb78/
2026-03-14 10:05:41.310359 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874cs2f2_e952e216-1adb-456a-9f7e-61d2dc2ddb78/extract/
2026-03-14 10:05:41.310376 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874cs2f2_e952e216-1adb-456a-9f7e-61d2dc2ddb78/extract/0.log
2026-03-14 10:05:41.310393 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874cs2f2_e952e216-1adb-456a-9f7e-61d2dc2ddb78/pull/
2026-03-14 10:05:41.310410 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874cs2f2_e952e216-1adb-456a-9f7e-61d2dc2ddb78/pull/0.log
2026-03-14 10:05:41.310431 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874cs2f2_e952e216-1adb-456a-9f7e-61d2dc2ddb78/util/
2026-03-14 10:05:41.310449 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874cs2f2_e952e216-1adb-456a-9f7e-61d2dc2ddb78/util/0.log
2026-03-14 10:05:41.310466 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c14m429_d916f254-7cc8-48b9-b01e-ba6270584eb3/
2026-03-14 10:05:41.310484 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c14m429_d916f254-7cc8-48b9-b01e-ba6270584eb3/extract/
2026-03-14 10:05:41.310500 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c14m429_d916f254-7cc8-48b9-b01e-ba6270584eb3/extract/0.log
2026-03-14 10:05:41.310517 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c14m429_d916f254-7cc8-48b9-b01e-ba6270584eb3/pull/
2026-03-14 10:05:41.310534 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c14m429_d916f254-7cc8-48b9-b01e-ba6270584eb3/pull/0.log
2026-03-14 10:05:41.310551 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c14m429_d916f254-7cc8-48b9-b01e-ba6270584eb3/util/
2026-03-14 10:05:41.310568 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c14m429_d916f254-7cc8-48b9-b01e-ba6270584eb3/util/0.log
2026-03-14 10:05:41.310586 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5qhxpx_bd5a3603-199d-45f6-80c7-afb96e29068f/
2026-03-14 10:05:41.310603 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5qhxpx_bd5a3603-199d-45f6-80c7-afb96e29068f/extract/
2026-03-14 10:05:41.310620 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5qhxpx_bd5a3603-199d-45f6-80c7-afb96e29068f/extract/0.log
2026-03-14 10:05:41.310637 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5qhxpx_bd5a3603-199d-45f6-80c7-afb96e29068f/pull/
2026-03-14 10:05:41.310654 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5qhxpx_bd5a3603-199d-45f6-80c7-afb96e29068f/pull/0.log
2026-03-14 10:05:41.310671 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5qhxpx_bd5a3603-199d-45f6-80c7-afb96e29068f/util/
2026-03-14 10:05:41.310709 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5qhxpx_bd5a3603-199d-45f6-80c7-afb96e29068f/util/0.log
2026-03-14 10:05:41.310728 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f0857cj7_cb06d762-fe54-4e6f-917e-1431b5948d9a/
2026-03-14 10:05:41.310750 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f0857cj7_cb06d762-fe54-4e6f-917e-1431b5948d9a/extract/
2026-03-14 10:05:41.310768 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f0857cj7_cb06d762-fe54-4e6f-917e-1431b5948d9a/extract/0.log
2026-03-14 10:05:41.310786 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f0857cj7_cb06d762-fe54-4e6f-917e-1431b5948d9a/pull/
2026-03-14 10:05:41.310803 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f0857cj7_cb06d762-fe54-4e6f-917e-1431b5948d9a/pull/0.log
2026-03-14 10:05:41.310820 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f0857cj7_cb06d762-fe54-4e6f-917e-1431b5948d9a/util/
2026-03-14 10:05:41.310837 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f0857cj7_cb06d762-fe54-4e6f-917e-1431b5948d9a/util/0.log
2026-03-14 10:05:41.310854 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-tt7cz_38ba05ff-31cc-4fa9-a6fc-e374e8c6ede6/
2026-03-14 10:05:41.310872 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-tt7cz_38ba05ff-31cc-4fa9-a6fc-e374e8c6ede6/extract-content/
2026-03-14 10:05:41.310889 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-tt7cz_38ba05ff-31cc-4fa9-a6fc-e374e8c6ede6/extract-content/0.log
2026-03-14 10:05:41.310905 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-tt7cz_38ba05ff-31cc-4fa9-a6fc-e374e8c6ede6/extract-utilities/
2026-03-14 10:05:41.310923 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-tt7cz_38ba05ff-31cc-4fa9-a6fc-e374e8c6ede6/extract-utilities/0.log
2026-03-14 10:05:41.310940 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-tt7cz_38ba05ff-31cc-4fa9-a6fc-e374e8c6ede6/registry-server/
2026-03-14 10:05:41.310956 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-tt7cz_38ba05ff-31cc-4fa9-a6fc-e374e8c6ede6/registry-server/0.log.gz
2026-03-14 10:05:41.310973 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-llm5h_353b52a0-aebf-426a-9006-74c1d24e8c48/
2026-03-14 10:05:41.310991 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-llm5h_353b52a0-aebf-426a-9006-74c1d24e8c48/extract-content/
2026-03-14 10:05:41.311008 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-llm5h_353b52a0-aebf-426a-9006-74c1d24e8c48/extract-content/0.log
2026-03-14 10:05:41.311025 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-llm5h_353b52a0-aebf-426a-9006-74c1d24e8c48/extract-utilities/
2026-03-14 10:05:41.311042 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-llm5h_353b52a0-aebf-426a-9006-74c1d24e8c48/extract-utilities/0.log
2026-03-14 10:05:41.311080 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-llm5h_353b52a0-aebf-426a-9006-74c1d24e8c48/registry-server/
2026-03-14 10:05:41.311103 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-llm5h_353b52a0-aebf-426a-9006-74c1d24e8c48/registry-server/0.log.gz
2026-03-14 10:05:41.311122 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_marketplace-operator-79b997595-565tx_3bd512f7-d36b-43b0-94af-cee8982d7630/
2026-03-14 10:05:41.311140 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_marketplace-operator-79b997595-565tx_3bd512f7-d36b-43b0-94af-cee8982d7630/marketplace-operator/
2026-03-14 10:05:41.311159 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_marketplace-operator-79b997595-565tx_3bd512f7-d36b-43b0-94af-cee8982d7630/marketplace-operator/0.log
2026-03-14 10:05:41.311177 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-tfbnl_166fa22e-f961-4d02-bd2a-a6e215204f4b/
2026-03-14 10:05:41.311195 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-tfbnl_166fa22e-f961-4d02-bd2a-a6e215204f4b/extract-content/
2026-03-14 10:05:41.311213 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-tfbnl_166fa22e-f961-4d02-bd2a-a6e215204f4b/extract-content/0.log
2026-03-14 10:05:41.311231 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-tfbnl_166fa22e-f961-4d02-bd2a-a6e215204f4b/extract-utilities/
2026-03-14 10:05:41.311249 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-tfbnl_166fa22e-f961-4d02-bd2a-a6e215204f4b/extract-utilities/0.log
2026-03-14 10:05:41.311267 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-tfbnl_166fa22e-f961-4d02-bd2a-a6e215204f4b/registry-server/
2026-03-14 10:05:41.311285 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-tfbnl_166fa22e-f961-4d02-bd2a-a6e215204f4b/registry-server/0.log
2026-03-14 10:05:41.311303 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-vx4kc_f9c9423f-f5aa-4a79-8084-f775e9a8af97/
2026-03-14 10:05:41.311320 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-vx4kc_f9c9423f-f5aa-4a79-8084-f775e9a8af97/extract-content/
2026-03-14 10:05:41.311337 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-vx4kc_f9c9423f-f5aa-4a79-8084-f775e9a8af97/extract-content/0.log
2026-03-14 10:05:41.311354 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-vx4kc_f9c9423f-f5aa-4a79-8084-f775e9a8af97/extract-utilities/
2026-03-14 10:05:41.311371 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-vx4kc_f9c9423f-f5aa-4a79-8084-f775e9a8af97/extract-utilities/0.log
2026-03-14 10:05:41.311389 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-vx4kc_f9c9423f-f5aa-4a79-8084-f775e9a8af97/registry-server/
2026-03-14 10:05:41.311405 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-vx4kc_f9c9423f-f5aa-4a79-8084-f775e9a8af97/registry-server/0.log.gz
2026-03-14 10:05:41.311423 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-p8t7w_da3d426b-76cf-40bd-8368-f52b1f74e841/
2026-03-14 10:05:41.311440 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-p8t7w_da3d426b-76cf-40bd-8368-f52b1f74e841/bond-cni-plugin/
2026-03-14 10:05:41.311462 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-p8t7w_da3d426b-76cf-40bd-8368-f52b1f74e841/bond-cni-plugin/0.log
2026-03-14 10:05:41.311592 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-p8t7w_da3d426b-76cf-40bd-8368-f52b1f74e841/cni-plugins/
2026-03-14 10:05:41.311619 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-p8t7w_da3d426b-76cf-40bd-8368-f52b1f74e841/cni-plugins/0.log
2026-03-14 10:05:41.311638 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-p8t7w_da3d426b-76cf-40bd-8368-f52b1f74e841/egress-router-binary-copy/
2026-03-14 10:05:41.311657 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-p8t7w_da3d426b-76cf-40bd-8368-f52b1f74e841/egress-router-binary-copy/0.log
2026-03-14 10:05:41.311674 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-p8t7w_da3d426b-76cf-40bd-8368-f52b1f74e841/kube-multus-additional-cni-plugins/
2026-03-14 10:05:41.311718 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-p8t7w_da3d426b-76cf-40bd-8368-f52b1f74e841/kube-multus-additional-cni-plugins/0.log
2026-03-14 10:05:41.311736 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-p8t7w_da3d426b-76cf-40bd-8368-f52b1f74e841/routeoverride-cni/
2026-03-14 10:05:41.311754 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-p8t7w_da3d426b-76cf-40bd-8368-f52b1f74e841/routeoverride-cni/0.log
2026-03-14 10:05:41.311771 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-p8t7w_da3d426b-76cf-40bd-8368-f52b1f74e841/whereabouts-cni-bincopy/
2026-03-14 10:05:41.311788 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-p8t7w_da3d426b-76cf-40bd-8368-f52b1f74e841/whereabouts-cni-bincopy/0.log
2026-03-14 10:05:41.311806 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-p8t7w_da3d426b-76cf-40bd-8368-f52b1f74e841/whereabouts-cni/
2026-03-14 10:05:41.311823 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-p8t7w_da3d426b-76cf-40bd-8368-f52b1f74e841/whereabouts-cni/0.log
2026-03-14 10:05:41.311840 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-admission-controller-857f4d67dd-bfc6h_9fafc3ce-d8b1-4179-be51-d7d3f0c17217/
2026-03-14 10:05:41.311858 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-admission-controller-857f4d67dd-bfc6h_9fafc3ce-d8b1-4179-be51-d7d3f0c17217/kube-rbac-proxy/
2026-03-14 10:05:41.311876 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-admission-controller-857f4d67dd-bfc6h_9fafc3ce-d8b1-4179-be51-d7d3f0c17217/kube-rbac-proxy/0.log
2026-03-14 10:05:41.311893 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-admission-controller-857f4d67dd-bfc6h_9fafc3ce-d8b1-4179-be51-d7d3f0c17217/multus-admission-controller/
2026-03-14 10:05:41.311910 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-admission-controller-857f4d67dd-bfc6h_9fafc3ce-d8b1-4179-be51-d7d3f0c17217/multus-admission-controller/0.log
2026-03-14 10:05:41.311932 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-sd9qb_32cf92ff-937a-41d1-b27a-127bd550a1eb/
2026-03-14 10:05:41.311950 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-sd9qb_32cf92ff-937a-41d1-b27a-127bd550a1eb/kube-multus/
2026-03-14 10:05:41.311967 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-sd9qb_32cf92ff-937a-41d1-b27a-127bd550a1eb/kube-multus/0.log
2026-03-14 10:05:41.311985 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-sd9qb_32cf92ff-937a-41d1-b27a-127bd550a1eb/kube-multus/1.log
2026-03-14 10:05:41.312003 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_network-metrics-daemon-2zndz_ca968925-6d0f-46b7-b12b-7c617faabd3c/
2026-03-14 10:05:41.312020 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_network-metrics-daemon-2zndz_ca968925-6d0f-46b7-b12b-7c617faabd3c/kube-rbac-proxy/
2026-03-14 10:05:41.312037 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_network-metrics-daemon-2zndz_ca968925-6d0f-46b7-b12b-7c617faabd3c/kube-rbac-proxy/0.log
2026-03-14 10:05:41.312054 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_network-metrics-daemon-2zndz_ca968925-6d0f-46b7-b12b-7c617faabd3c/network-metrics-daemon/
2026-03-14 10:05:41.312070 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_network-metrics-daemon-2zndz_ca968925-6d0f-46b7-b12b-7c617faabd3c/network-metrics-daemon/0.log
2026-03-14 10:05:41.312088 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-must-gather-s74vk_must-gather-88lrc_4c313e64-65bc-4dfd-a1cf-61956ec5870e/
2026-03-14 10:05:41.312104 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-must-gather-s74vk_must-gather-88lrc_4c313e64-65bc-4dfd-a1cf-61956ec5870e/copy/
2026-03-14 10:05:41.312122 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-must-gather-s74vk_must-gather-88lrc_4c313e64-65bc-4dfd-a1cf-61956ec5870e/gather/
2026-03-14 10:05:41.312140 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-console_networking-console-plugin-85b44fc459-gdk6g_5fe485a1-e14f-4c09-b5b9-f252bc42b7e8/
2026-03-14 10:05:41.312157 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-console_networking-console-plugin-85b44fc459-gdk6g_5fe485a1-e14f-4c09-b5b9-f252bc42b7e8/networking-console-plugin/
2026-03-14 10:05:41.312174 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-console_networking-console-plugin-85b44fc459-gdk6g_5fe485a1-e14f-4c09-b5b9-f252bc42b7e8/networking-console-plugin/0.log
2026-03-14 10:05:41.312192 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-source-55646444c4-trplf_9d751cbb-f2e2-430d-9754-c882a5e924a5/
2026-03-14 10:05:41.312209 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-source-55646444c4-trplf_9d751cbb-f2e2-430d-9754-c882a5e924a5/check-endpoints/
2026-03-14 10:05:41.312226 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-source-55646444c4-trplf_9d751cbb-f2e2-430d-9754-c882a5e924a5/check-endpoints/0.log
2026-03-14 10:05:41.312243 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-target-xd92c_3b6479f0-333b-4a96-9adf-2099afdc2447/
2026-03-14 10:05:41.312260 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-target-xd92c_3b6479f0-333b-4a96-9adf-2099afdc2447/network-check-target-container/
2026-03-14 10:05:41.312281 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-target-xd92c_3b6479f0-333b-4a96-9adf-2099afdc2447/network-check-target-container/0.log
2026-03-14 10:05:41.312299 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-node-identity_network-node-identity-vrzqb_ef543e1b-8068-4ea3-b32a-61027b32e95d/
2026-03-14 10:05:41.312316 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-node-identity_network-node-identity-vrzqb_ef543e1b-8068-4ea3-b32a-61027b32e95d/approver/
2026-03-14 10:05:41.312333 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-node-identity_network-node-identity-vrzqb_ef543e1b-8068-4ea3-b32a-61027b32e95d/approver/0.log
2026-03-14 10:05:41.312349 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-node-identity_network-node-identity-vrzqb_ef543e1b-8068-4ea3-b32a-61027b32e95d/webhook/
2026-03-14 10:05:41.312366 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-node-identity_network-node-identity-vrzqb_ef543e1b-8068-4ea3-b32a-61027b32e95d/webhook/0.log
2026-03-14 10:05:41.312383 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_iptables-alerter-4ln5h_d75a4c96-2883-4a0b-bab2-0fab2b6c0b49/
2026-03-14 10:05:41.312400 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_iptables-alerter-4ln5h_d75a4c96-2883-4a0b-bab2-0fab2b6c0b49/iptables-alerter/
2026-03-14 10:05:41.312417 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_iptables-alerter-4ln5h_d75a4c96-2883-4a0b-bab2-0fab2b6c0b49/iptables-alerter/0.log
2026-03-14 10:05:41.312434 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_network-operator-58b4c7f79c-55gtf_37a5e44f-9a88-4405-be8a-b645485e7312/
2026-03-14 10:05:41.312451 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_network-operator-58b4c7f79c-55gtf_37a5e44f-9a88-4405-be8a-b645485e7312/network-operator/
2026-03-14 10:05:41.312468 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_network-operator-58b4c7f79c-55gtf_37a5e44f-9a88-4405-be8a-b645485e7312/network-operator/0.log
2026-03-14 10:05:41.312484 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-console-plugin-86f58fcf4-wwth2_22c4bb4f-2852-4c58-a0db-17793ec80671/
2026-03-14 10:05:41.312505 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-console-plugin-86f58fcf4-wwth2_22c4bb4f-2852-4c58-a0db-17793ec80671/nmstate-console-plugin/
2026-03-14 10:05:41.312529 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-console-plugin-86f58fcf4-wwth2_22c4bb4f-2852-4c58-a0db-17793ec80671/nmstate-console-plugin/0.log
2026-03-14 10:05:41.312552 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-handler-fqs5c_951a69ab-6e2a-4ff3-bcf3-db69fbfcb349/
2026-03-14 10:05:41.312570 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-handler-fqs5c_951a69ab-6e2a-4ff3-bcf3-db69fbfcb349/nmstate-handler/
2026-03-14 10:05:41.312587 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-handler-fqs5c_951a69ab-6e2a-4ff3-bcf3-db69fbfcb349/nmstate-handler/0.log
2026-03-14 10:05:41.312605 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-metrics-9b8c8685d-bcwrg_c108acb5-004b-4c4e-b8e9-44aed45a2b86/
2026-03-14 10:05:41.312627 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-metrics-9b8c8685d-bcwrg_c108acb5-004b-4c4e-b8e9-44aed45a2b86/kube-rbac-proxy/
2026-03-14 10:05:41.312645 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-metrics-9b8c8685d-bcwrg_c108acb5-004b-4c4e-b8e9-44aed45a2b86/kube-rbac-proxy/0.log
2026-03-14 10:05:41.312662 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-metrics-9b8c8685d-bcwrg_c108acb5-004b-4c4e-b8e9-44aed45a2b86/nmstate-metrics/
2026-03-14 10:05:41.312699 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-metrics-9b8c8685d-bcwrg_c108acb5-004b-4c4e-b8e9-44aed45a2b86/nmstate-metrics/0.log
2026-03-14 10:05:41.312720 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-operator-796d4cfff4-n7k5d_63096d1d-7063-4821-b735-e3e65e2b9f64/
2026-03-14 10:05:41.312736 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-operator-796d4cfff4-n7k5d_63096d1d-7063-4821-b735-e3e65e2b9f64/nmstate-operator/
2026-03-14 10:05:41.312754 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-operator-796d4cfff4-n7k5d_63096d1d-7063-4821-b735-e3e65e2b9f64/nmstate-operator/0.log
2026-03-14 10:05:41.312770 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-webhook-5f558f5558-hcdc6_185eedfb-940a-4fb8-943f-3f2761f44301/
2026-03-14 10:05:41.312787 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-webhook-5f558f5558-hcdc6_185eedfb-940a-4fb8-943f-3f2761f44301/nmstate-webhook/
2026-03-14 10:05:41.312804 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-webhook-5f558f5558-hcdc6_185eedfb-940a-4fb8-943f-3f2761f44301/nmstate-webhook/0.log
2026-03-14 10:05:41.312821 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-oauth-apiserver_apiserver-7bbb656c7d-4ftzb_1724971d-d75c-4e8c-a951-0f10e9c5e799/
2026-03-14 10:05:41.312838 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-oauth-apiserver_apiserver-7bbb656c7d-4ftzb_1724971d-d75c-4e8c-a951-0f10e9c5e799/fix-audit-permissions/
2026-03-14 10:05:41.312855 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-oauth-apiserver_apiserver-7bbb656c7d-4ftzb_1724971d-d75c-4e8c-a951-0f10e9c5e799/fix-audit-permissions/0.log
2026-03-14 10:05:41.312872 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-oauth-apiserver_apiserver-7bbb656c7d-4ftzb_1724971d-d75c-4e8c-a951-0f10e9c5e799/oauth-apiserver/
2026-03-14 10:05:41.312889 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-oauth-apiserver_apiserver-7bbb656c7d-4ftzb_1724971d-d75c-4e8c-a951-0f10e9c5e799/oauth-apiserver/0.log
2026-03-14 10:05:41.312905 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_catalog-operator-68c6474976-z45r5_c4b10c60-ed92-4bcf-9385-bf169e81051f/
2026-03-14 10:05:41.312923 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_catalog-operator-68c6474976-z45r5_c4b10c60-ed92-4bcf-9385-bf169e81051f/catalog-operator/
2026-03-14 10:05:41.312940 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_catalog-operator-68c6474976-z45r5_c4b10c60-ed92-4bcf-9385-bf169e81051f/catalog-operator/0.log.gz
2026-03-14 10:05:41.312957 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29558010-l6rht_e0fce781-5334-45d5-83be-69a184093680/
2026-03-14 10:05:41.312974 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29558010-l6rht_e0fce781-5334-45d5-83be-69a184093680/collect-profiles/
2026-03-14 10:05:41.312996 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29558010-l6rht_e0fce781-5334-45d5-83be-69a184093680/collect-profiles/0.log
2026-03-14 10:05:41.313013 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29558025-9wsdz_b616c8e5-ade8-4e14-bf70-3d1887901ced/
2026-03-14 10:05:41.313030 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29558025-9wsdz_b616c8e5-ade8-4e14-bf70-3d1887901ced/collect-profiles/
2026-03-14 10:05:41.313048 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29558025-9wsdz_b616c8e5-ade8-4e14-bf70-3d1887901ced/collect-profiles/0.log
2026-03-14 10:05:41.313065 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29558040-7nf9m_f9071be7-7dbc-4e12-bb11-1b6228c57534/
2026-03-14 10:05:41.313082 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29558040-7nf9m_f9071be7-7dbc-4e12-bb11-1b6228c57534/collect-profiles/
2026-03-14 10:05:41.313099 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29558040-7nf9m_f9071be7-7dbc-4e12-bb11-1b6228c57534/collect-profiles/0.log
2026-03-14 10:05:41.313116 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_olm-operator-6b444d44fb-29xcc_e494d22d-e0ff-4fab-85d6-5241d72ff039/
2026-03-14 10:05:41.313134 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_olm-operator-6b444d44fb-29xcc_e494d22d-e0ff-4fab-85d6-5241d72ff039/olm-operator/
2026-03-14 10:05:41.313151 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_olm-operator-6b444d44fb-29xcc_e494d22d-e0ff-4fab-85d6-5241d72ff039/olm-operator/0.log
2026-03-14 10:05:41.313168 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_package-server-manager-789f6589d5-hc4vf_694288bd-c353-44f3-806b-5ce7c6a10486/
2026-03-14 10:05:41.313185 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_package-server-manager-789f6589d5-hc4vf_694288bd-c353-44f3-806b-5ce7c6a10486/kube-rbac-proxy/
2026-03-14 10:05:41.313201 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_package-server-manager-789f6589d5-hc4vf_694288bd-c353-44f3-806b-5ce7c6a10486/kube-rbac-proxy/0.log
2026-03-14 10:05:41.313218 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_package-server-manager-789f6589d5-hc4vf_694288bd-c353-44f3-806b-5ce7c6a10486/package-server-manager/
2026-03-14 10:05:41.313235 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_package-server-manager-789f6589d5-hc4vf_694288bd-c353-44f3-806b-5ce7c6a10486/package-server-manager/0.log
2026-03-14 10:05:41.313254 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_packageserver-d55dfcdfc-f5gt8_8278bb86-cd39-4ecb-b0c6-75ec86c0763a/
2026-03-14 10:05:41.313273 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_packageserver-d55dfcdfc-f5gt8_8278bb86-cd39-4ecb-b0c6-75ec86c0763a/packageserver/
2026-03-14 10:05:41.313290 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_packageserver-d55dfcdfc-f5gt8_8278bb86-cd39-4ecb-b0c6-75ec86c0763a/packageserver/0.log
2026-03-14 10:05:41.313311 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-68bc856cb9-7vsx5_7f706824-088d-4d70-830e-4844447b900d/
2026-03-14 10:05:41.313329 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-68bc856cb9-7vsx5_7f706824-088d-4d70-830e-4844447b900d/prometheus-operator/
2026-03-14 10:05:41.313346 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-68bc856cb9-7vsx5_7f706824-088d-4d70-830e-4844447b900d/prometheus-operator/0.log
2026-03-14 10:05:41.313363 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-69fb744b97-5knkk_7d01a80d-2445-472e-ab28-81703fb66a27/
2026-03-14 10:05:41.313425 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-69fb744b97-5knkk_7d01a80d-2445-472e-ab28-81703fb66a27/prometheus-operator-admission-webhook/
2026-03-14 10:05:41.313448 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-69fb744b97-5knkk_7d01a80d-2445-472e-ab28-81703fb66a27/prometheus-operator-admission-webhook/0.log
2026-03-14 10:05:41.313466 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-69fb744b97-scrd4_35d96605-ef77-45e3-b12c-26c192a2dd50/
2026-03-14 10:05:41.313483 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-69fb744b97-scrd4_35d96605-ef77-45e3-b12c-26c192a2dd50/prometheus-operator-admission-webhook/
2026-03-14 10:05:41.313500 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-69fb744b97-scrd4_35d96605-ef77-45e3-b12c-26c192a2dd50/prometheus-operator-admission-webhook/0.log
2026-03-14 10:05:41.313518 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-operator-59bdc8b94-7l9bs_d8bb5f31-a201-4fa9-a297-f294c2aefe07/
2026-03-14 10:05:41.313535 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-operator-59bdc8b94-7l9bs_d8bb5f31-a201-4fa9-a297-f294c2aefe07/operator/
2026-03-14 10:05:41.313552 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-operator-59bdc8b94-7l9bs_d8bb5f31-a201-4fa9-a297-f294c2aefe07/operator/0.log
2026-03-14 10:05:41.313568 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-ui-dashboards-66cbf594b5-k8bb7_2c318b20-fdd8-4ebb-817b-14f94d14580c/
2026-03-14 10:05:41.313585 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-ui-dashboards-66cbf594b5-k8bb7_2c318b20-fdd8-4ebb-817b-14f94d14580c/observability-ui-dashboards/
2026-03-14 10:05:41.313602 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-ui-dashboards-66cbf594b5-k8bb7_2c318b20-fdd8-4ebb-817b-14f94d14580c/observability-ui-dashboards/0.log
2026-03-14 10:05:41.313619 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_perses-operator-5bf474d74f-8glvj_5e9fbec7-6a0e-48cf-95fb-5a470a77ff4c/
2026-03-14 10:05:41.313636 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_perses-operator-5bf474d74f-8glvj_5e9fbec7-6a0e-48cf-95fb-5a470a77ff4c/perses-operator/
2026-03-14 10:05:41.313653 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_perses-operator-5bf474d74f-8glvj_5e9fbec7-6a0e-48cf-95fb-5a470a77ff4c/perses-operator/0.log
2026-03-14 10:05:41.313688 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-control-plane-749d76644c-lqxth_70117c20-1b22-4e7e-a663-1665e5b64a0f/
2026-03-14 10:05:41.313711 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-control-plane-749d76644c-lqxth_70117c20-1b22-4e7e-a663-1665e5b64a0f/kube-rbac-proxy/
2026-03-14 10:05:41.313730 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-control-plane-749d76644c-lqxth_70117c20-1b22-4e7e-a663-1665e5b64a0f/kube-rbac-proxy/0.log
2026-03-14 10:05:41.313747 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-control-plane-749d76644c-lqxth_70117c20-1b22-4e7e-a663-1665e5b64a0f/ovnkube-cluster-manager/
2026-03-14 10:05:41.313764 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-control-plane-749d76644c-lqxth_70117c20-1b22-4e7e-a663-1665e5b64a0f/ovnkube-cluster-manager/0.log
2026-03-14 10:05:41.313781 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-rqzbl_bc4715f8-9fa2-474e-b1cd-94ceeeff9505/
2026-03-14 10:05:41.313799 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-rqzbl_bc4715f8-9fa2-474e-b1cd-94ceeeff9505/kube-rbac-proxy-node/
2026-03-14 10:05:41.313815 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-rqzbl_bc4715f8-9fa2-474e-b1cd-94ceeeff9505/kube-rbac-proxy-node/0.log
2026-03-14 10:05:41.313832 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-rqzbl_bc4715f8-9fa2-474e-b1cd-94ceeeff9505/kube-rbac-proxy-ovn-metrics/
2026-03-14 10:05:41.313848 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-rqzbl_bc4715f8-9fa2-474e-b1cd-94ceeeff9505/kube-rbac-proxy-ovn-metrics/0.log
2026-03-14 10:05:41.313865 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-rqzbl_bc4715f8-9fa2-474e-b1cd-94ceeeff9505/kubecfg-setup/
2026-03-14 10:05:41.313883 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-rqzbl_bc4715f8-9fa2-474e-b1cd-94ceeeff9505/kubecfg-setup/0.log
2026-03-14 10:05:41.313898 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-rqzbl_bc4715f8-9fa2-474e-b1cd-94ceeeff9505/nbdb/
2026-03-14 10:05:41.313915 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-rqzbl_bc4715f8-9fa2-474e-b1cd-94ceeeff9505/nbdb/0.log
2026-03-14 10:05:41.313932 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-rqzbl_bc4715f8-9fa2-474e-b1cd-94ceeeff9505/northd/
2026-03-14 10:05:41.313949 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-rqzbl_bc4715f8-9fa2-474e-b1cd-94ceeeff9505/northd/0.log
2026-03-14 10:05:41.313965 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-rqzbl_bc4715f8-9fa2-474e-b1cd-94ceeeff9505/ovn-acl-logging/
2026-03-14 10:05:41.313981 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-rqzbl_bc4715f8-9fa2-474e-b1cd-94ceeeff9505/ovn-acl-logging/0.log
2026-03-14 10:05:41.313997 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-rqzbl_bc4715f8-9fa2-474e-b1cd-94ceeeff9505/ovn-controller/
2026-03-14 10:05:41.314018 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-rqzbl_bc4715f8-9fa2-474e-b1cd-94ceeeff9505/ovn-controller/0.log
2026-03-14 10:05:41.314035 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-rqzbl_bc4715f8-9fa2-474e-b1cd-94ceeeff9505/ovnkube-controller/
2026-03-14 10:05:41.314051 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-rqzbl_bc4715f8-9fa2-474e-b1cd-94ceeeff9505/ovnkube-controller/0.log.gz
2026-03-14 10:05:41.314068 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-rqzbl_bc4715f8-9fa2-474e-b1cd-94ceeeff9505/sbdb/
2026-03-14 10:05:41.314085 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-rqzbl_bc4715f8-9fa2-474e-b1cd-94ceeeff9505/sbdb/0.log
2026-03-14 10:05:41.314101 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-route-controller-manager_route-controller-manager-59954f54c6-bnvq9_b25c6fc4-54d8-4412-95b1-cf5f7e2f01ae/
2026-03-14 10:05:41.314118 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-route-controller-manager_route-controller-manager-59954f54c6-bnvq9_b25c6fc4-54d8-4412-95b1-cf5f7e2f01ae/route-controller-manager/
2026-03-14 10:05:41.314135 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-route-controller-manager_route-controller-manager-59954f54c6-bnvq9_b25c6fc4-54d8-4412-95b1-cf5f7e2f01ae/route-controller-manager/0.log
2026-03-14 10:05:41.314152 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca-operator_service-ca-operator-777779d784-dtnxl_e037c68c-36ae-47b2-b3f5-e159c512cf72/
2026-03-14 10:05:41.314168 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca-operator_service-ca-operator-777779d784-dtnxl_e037c68c-36ae-47b2-b3f5-e159c512cf72/service-ca-operator/
2026-03-14 10:05:41.314185 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca-operator_service-ca-operator-777779d784-dtnxl_e037c68c-36ae-47b2-b3f5-e159c512cf72/service-ca-operator/0.log
2026-03-14 10:05:41.314206 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca_service-ca-9c57cc56f-zldnt_dcf6a857-49d2-4319-b083-3ff05a19191b/
2026-03-14 10:05:41.314226 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca_service-ca-9c57cc56f-zldnt_dcf6a857-49d2-4319-b083-3ff05a19191b/service-ca-controller/
2026-03-14 10:05:41.314243 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca_service-ca-9c57cc56f-zldnt_dcf6a857-49d2-4319-b083-3ff05a19191b/service-ca-controller/0.log
2026-03-14 10:05:41.314260 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_65270e5b79eb73ea65900207edd02f6e862f5863f7b0ae98982f030d4cpkw4d_8cd388c8-6f9c-469d-b08f-32ad35850e12/
2026-03-14 10:05:41.314276 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_65270e5b79eb73ea65900207edd02f6e862f5863f7b0ae98982f030d4cpkw4d_8cd388c8-6f9c-469d-b08f-32ad35850e12/extract/
2026-03-14 10:05:41.314293 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_65270e5b79eb73ea65900207edd02f6e862f5863f7b0ae98982f030d4cpkw4d_8cd388c8-6f9c-469d-b08f-32ad35850e12/extract/0.log
2026-03-14 10:05:41.314309 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_65270e5b79eb73ea65900207edd02f6e862f5863f7b0ae98982f030d4cpkw4d_8cd388c8-6f9c-469d-b08f-32ad35850e12/pull/
2026-03-14 10:05:41.314358 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_65270e5b79eb73ea65900207edd02f6e862f5863f7b0ae98982f030d4cpkw4d_8cd388c8-6f9c-469d-b08f-32ad35850e12/pull/0.log
2026-03-14 10:05:41.314385 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_65270e5b79eb73ea65900207edd02f6e862f5863f7b0ae98982f030d4cpkw4d_8cd388c8-6f9c-469d-b08f-32ad35850e12/util/
2026-03-14 10:05:41.314403 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_65270e5b79eb73ea65900207edd02f6e862f5863f7b0ae98982f030d4cpkw4d_8cd388c8-6f9c-469d-b08f-32ad35850e12/util/0.log
2026-03-14 10:05:41.314420 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_b2fc087f8a76dbce8476595fc384d33288943d0097430396a1362c765eshgz7_9892ab03-eb13-4560-a688-5e38fd015478/
2026-03-14 10:05:41.314438 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_b2fc087f8a76dbce8476595fc384d33288943d0097430396a1362c765eshgz7_9892ab03-eb13-4560-a688-5e38fd015478/extract/
2026-03-14 10:05:41.314504 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_b2fc087f8a76dbce8476595fc384d33288943d0097430396a1362c765eshgz7_9892ab03-eb13-4560-a688-5e38fd015478/extract/0.log
2026-03-14 10:05:41.314533 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_b2fc087f8a76dbce8476595fc384d33288943d0097430396a1362c765eshgz7_9892ab03-eb13-4560-a688-5e38fd015478/pull/
2026-03-14 10:05:41.314552 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_b2fc087f8a76dbce8476595fc384d33288943d0097430396a1362c765eshgz7_9892ab03-eb13-4560-a688-5e38fd015478/pull/0.log
2026-03-14 10:05:41.314569 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_b2fc087f8a76dbce8476595fc384d33288943d0097430396a1362c765eshgz7_9892ab03-eb13-4560-a688-5e38fd015478/util/
2026-03-14 10:05:41.314587 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_b2fc087f8a76dbce8476595fc384d33288943d0097430396a1362c765eshgz7_9892ab03-eb13-4560-a688-5e38fd015478/util/0.log
2026-03-14 10:05:41.314604 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_barbican-operator-controller-manager-d47688694-ktr4z_09b1d07d-69b7-427b-9577-2a782637a2b1/
2026-03-14 10:05:41.314621 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_barbican-operator-controller-manager-d47688694-ktr4z_09b1d07d-69b7-427b-9577-2a782637a2b1/manager/
2026-03-14 10:05:41.314637 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_barbican-operator-controller-manager-d47688694-ktr4z_09b1d07d-69b7-427b-9577-2a782637a2b1/manager/0.log
2026-03-14 10:05:41.314654 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_cinder-operator-controller-manager-984cd4dcf-75kt6_0fbf4f22-f279-4841-b829-6d35d2dd0bfc/
2026-03-14 10:05:41.314671 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_cinder-operator-controller-manager-984cd4dcf-75kt6_0fbf4f22-f279-4841-b829-6d35d2dd0bfc/manager/
2026-03-14 10:05:41.314707 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_cinder-operator-controller-manager-984cd4dcf-75kt6_0fbf4f22-f279-4841-b829-6d35d2dd0bfc/manager/0.log.gz
2026-03-14 10:05:41.314726 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_designate-operator-controller-manager-66d56f6ff4-mxn5j_0e6cdd9e-39ed-4359-9b74-54e23a0c70e2/
2026-03-14 10:05:41.314743 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_designate-operator-controller-manager-66d56f6ff4-mxn5j_0e6cdd9e-39ed-4359-9b74-54e23a0c70e2/manager/
2026-03-14 10:05:41.314760 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_designate-operator-controller-manager-66d56f6ff4-mxn5j_0e6cdd9e-39ed-4359-9b74-54e23a0c70e2/manager/0.log
2026-03-14 10:05:41.314782 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_glance-operator-controller-manager-5964f64c48-g29z7_354f7562-fd32-491f-a3dd-b42bcf7d30cb/
2026-03-14 10:05:41.314800 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_glance-operator-controller-manager-5964f64c48-g29z7_354f7562-fd32-491f-a3dd-b42bcf7d30cb/manager/
2026-03-14 10:05:41.314818 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_glance-operator-controller-manager-5964f64c48-g29z7_354f7562-fd32-491f-a3dd-b42bcf7d30cb/manager/0.log
2026-03-14 10:05:41.314836 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_heat-operator-controller-manager-77b6666d85-czghc_7d4e28ff-8c1b-4533-a733-97c44a2b7b0a/
2026-03-14 10:05:41.314852 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_heat-operator-controller-manager-77b6666d85-czghc_7d4e28ff-8c1b-4533-a733-97c44a2b7b0a/manager/
2026-03-14 10:05:41.314869 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_heat-operator-controller-manager-77b6666d85-czghc_7d4e28ff-8c1b-4533-a733-97c44a2b7b0a/manager/0.log
2026-03-14 10:05:41.314886 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_horizon-operator-controller-manager-6d9d6b584d-bmb27_3da8acd2-64eb-4f3f-b28e-80d5bb4e4743/
2026-03-14 10:05:41.314902 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_horizon-operator-controller-manager-6d9d6b584d-bmb27_3da8acd2-64eb-4f3f-b28e-80d5bb4e4743/manager/
2026-03-14 10:05:41.314919 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_horizon-operator-controller-manager-6d9d6b584d-bmb27_3da8acd2-64eb-4f3f-b28e-80d5bb4e4743/manager/0.log
2026-03-14 10:05:41.314936 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_infra-operator-controller-manager-54dc5b8f8d-ffxmw_46ec1c5f-0334-4d5d-b16a-8e01ffc76613/
2026-03-14 10:05:41.314952 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_infra-operator-controller-manager-54dc5b8f8d-ffxmw_46ec1c5f-0334-4d5d-b16a-8e01ffc76613/manager/
2026-03-14 10:05:41.314969 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_infra-operator-controller-manager-54dc5b8f8d-ffxmw_46ec1c5f-0334-4d5d-b16a-8e01ffc76613/manager/0.log
2026-03-14 10:05:41.314986 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ironic-operator-controller-manager-5bc894d9b-8wvhp_47f2d7ba-aac2-44f4-9146-ca20192a08ab/
2026-03-14 10:05:41.315003 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ironic-operator-controller-manager-5bc894d9b-8wvhp_47f2d7ba-aac2-44f4-9146-ca20192a08ab/manager/
2026-03-14 10:05:41.315020 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ironic-operator-controller-manager-5bc894d9b-8wvhp_47f2d7ba-aac2-44f4-9146-ca20192a08ab/manager/0.log
2026-03-14 10:05:41.315037 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_keystone-operator-controller-manager-684f77d66d-7v8h2_d4d69189-dfff-4f22-b978-4988c505df47/
2026-03-14 10:05:41.315065 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_keystone-operator-controller-manager-684f77d66d-7v8h2_d4d69189-dfff-4f22-b978-4988c505df47/manager/
2026-03-14 10:05:41.315084 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_keystone-operator-controller-manager-684f77d66d-7v8h2_d4d69189-dfff-4f22-b978-4988c505df47/manager/0.log
2026-03-14 10:05:41.315114 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_manila-operator-controller-manager-57b484b4df-fqfbh_720b9936-b2c8-4b7b-ab8f-c8ecfda9052d/
2026-03-14 10:05:41.315132 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_manila-operator-controller-manager-57b484b4df-fqfbh_720b9936-b2c8-4b7b-ab8f-c8ecfda9052d/manager/
2026-03-14 10:05:41.315151 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_manila-operator-controller-manager-57b484b4df-fqfbh_720b9936-b2c8-4b7b-ab8f-c8ecfda9052d/manager/0.log
2026-03-14 10:05:41.315168 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_mariadb-operator-controller-manager-5b6b6b4c9f-pdpx8_ddd8e1af-7084-4685-96a9-ca4eaa9a4742/
2026-03-14 10:05:41.315185 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_mariadb-operator-controller-manager-5b6b6b4c9f-pdpx8_ddd8e1af-7084-4685-96a9-ca4eaa9a4742/manager/
2026-03-14 10:05:41.315202 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_mariadb-operator-controller-manager-5b6b6b4c9f-pdpx8_ddd8e1af-7084-4685-96a9-ca4eaa9a4742/manager/0.log
2026-03-14 10:05:41.315219 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_neutron-operator-controller-manager-776c5696bf-9dlkn_28f29a72-e196-4962-87b3-cc49b8591cc0/
2026-03-14 10:05:41.315236 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_neutron-operator-controller-manager-776c5696bf-9dlkn_28f29a72-e196-4962-87b3-cc49b8591cc0/manager/
2026-03-14 10:05:41.315253 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_neutron-operator-controller-manager-776c5696bf-9dlkn_28f29a72-e196-4962-87b3-cc49b8591cc0/manager/0.log
2026-03-14 10:05:41.315270 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_nova-operator-controller-manager-7f84474648-g59td_abe14da3-e8a8-45f6-aebb-5df03c384a35/
2026-03-14 10:05:41.315287 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_nova-operator-controller-manager-7f84474648-g59td_abe14da3-e8a8-45f6-aebb-5df03c384a35/manager/
2026-03-14 10:05:41.315303 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_nova-operator-controller-manager-7f84474648-g59td_abe14da3-e8a8-45f6-aebb-5df03c384a35/manager/0.log
2026-03-14 10:05:41.315320 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_octavia-operator-controller-manager-5f4f55cb5c-zrfcm_72b0bab9-36e7-49da-af7b-ff4fe9475545/
2026-03-14 10:05:41.315337 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_octavia-operator-controller-manager-5f4f55cb5c-zrfcm_72b0bab9-36e7-49da-af7b-ff4fe9475545/manager/
2026-03-14 10:05:41.315353 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_octavia-operator-controller-manager-5f4f55cb5c-zrfcm_72b0bab9-36e7-49da-af7b-ff4fe9475545/manager/0.log
2026-03-14 10:05:41.315370 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-baremetal-operator-controller-manager-557ccf57b7slhjt_f5a4b439-9571-4fdb-8796-3bd729646aa3/
2026-03-14 10:05:41.315387 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-baremetal-operator-controller-manager-557ccf57b7slhjt_f5a4b439-9571-4fdb-8796-3bd729646aa3/manager/
2026-03-14 10:05:41.315404 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-baremetal-operator-controller-manager-557ccf57b7slhjt_f5a4b439-9571-4fdb-8796-3bd729646aa3/manager/0.log
2026-03-14 10:05:41.315427 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-controller-manager-59cdb7596d-l4ggb_a0339dbb-a75e-465b-a30a-75c3091f506c/
2026-03-14 10:05:41.315445 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-controller-manager-59cdb7596d-l4ggb_a0339dbb-a75e-465b-a30a-75c3091f506c/manager/
2026-03-14 10:05:41.315462 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-controller-manager-59cdb7596d-l4ggb_a0339dbb-a75e-465b-a30a-75c3091f506c/manager/0.log.gz
2026-03-14 10:05:41.315478 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-index-6s6vv_8f507af6-440f-42c3-839b-0466d546a5a0/
2026-03-14 10:05:41.315495 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-index-6s6vv_8f507af6-440f-42c3-839b-0466d546a5a0/registry-server/
2026-03-14 10:05:41.315512 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-index-6s6vv_8f507af6-440f-42c3-839b-0466d546a5a0/registry-server/0.log
2026-03-14 10:05:41.315528 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ovn-operator-controller-manager-bbc5b68f9-xctxr_92eda9b1-e96f-4a71-93f8-a595c6f5117d/
2026-03-14 10:05:41.315544 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ovn-operator-controller-manager-bbc5b68f9-xctxr_92eda9b1-e96f-4a71-93f8-a595c6f5117d/manager/
2026-03-14 10:05:41.315560 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ovn-operator-controller-manager-bbc5b68f9-xctxr_92eda9b1-e96f-4a71-93f8-a595c6f5117d/manager/0.log
2026-03-14 10:05:41.315577 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_placement-operator-controller-manager-574d45c66c-tx8dt_bdefcc7b-402d-4c04-bc48-aa8b918d1b15/
2026-03-14 10:05:41.315594 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_placement-operator-controller-manager-574d45c66c-tx8dt_bdefcc7b-402d-4c04-bc48-aa8b918d1b15/manager/
2026-03-14 10:05:41.315610 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_placement-operator-controller-manager-574d45c66c-tx8dt_bdefcc7b-402d-4c04-bc48-aa8b918d1b15/manager/0.log
2026-03-14 10:05:41.315627 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_rabbitmq-cluster-operator-manager-668c99d594-qjs78_f75cfc62-a50d-4bad-9868-c58dc328cdbb/
2026-03-14 10:05:41.315644 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_rabbitmq-cluster-operator-manager-668c99d594-qjs78_f75cfc62-a50d-4bad-9868-c58dc328cdbb/operator/
2026-03-14 10:05:41.315661 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_rabbitmq-cluster-operator-manager-668c99d594-qjs78_f75cfc62-a50d-4bad-9868-c58dc328cdbb/operator/0.log
2026-03-14 10:05:41.315693 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_swift-operator-controller-manager-7f9cc5dd44-kbcdm_cf4074be-d697-47bf-a44c-57822afebb15/
2026-03-14 10:05:41.315715 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_swift-operator-controller-manager-7f9cc5dd44-kbcdm_cf4074be-d697-47bf-a44c-57822afebb15/manager/
2026-03-14 10:05:41.315733 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_swift-operator-controller-manager-7f9cc5dd44-kbcdm_cf4074be-d697-47bf-a44c-57822afebb15/manager/0.log
2026-03-14 10:05:41.315751 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_telemetry-operator-controller-manager-6854b8b9d9-vgstg_f1e0e04a-1194-4fa6-878f-318d360845d3/
2026-03-14 10:05:41.315772 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_telemetry-operator-controller-manager-6854b8b9d9-vgstg_f1e0e04a-1194-4fa6-878f-318d360845d3/manager/
2026-03-14 10:05:41.315789 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_telemetry-operator-controller-manager-6854b8b9d9-vgstg_f1e0e04a-1194-4fa6-878f-318d360845d3/manager/0.log.gz
2026-03-14 10:05:41.315806 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_test-operator-controller-manager-5c5cb9c4d7-9nkkk_99c2bbc7-1885-4031-9478-a839c36bfbc5/
2026-03-14 10:05:41.315823 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_test-operator-controller-manager-5c5cb9c4d7-9nkkk_99c2bbc7-1885-4031-9478-a839c36bfbc5/manager/
2026-03-14 10:05:41.315839 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_test-operator-controller-manager-5c5cb9c4d7-9nkkk_99c2bbc7-1885-4031-9478-a839c36bfbc5/manager/0.log
2026-03-14 10:05:41.315855 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-controller-manager-66c89757b4-kgcsg_8b453c13-357e-4996-95ba-02e6d834a5b8/
2026-03-14 10:05:41.315872 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-controller-manager-66c89757b4-kgcsg_8b453c13-357e-4996-95ba-02e6d834a5b8/manager/
2026-03-14 10:05:41.315889 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-controller-manager-66c89757b4-kgcsg_8b453c13-357e-4996-95ba-02e6d834a5b8/manager/0.log.gz
2026-03-14 10:05:41.315906 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-index-7zk59_fa363db9-1707-497d-afab-dcf88242cdc2/
2026-03-14 10:05:41.315923 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-index-7zk59_fa363db9-1707-497d-afab-dcf88242cdc2/registry-server/
2026-03-14 10:05:41.315940 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-index-7zk59_fa363db9-1707-497d-afab-dcf88242cdc2/registry-server/0.log
2026-03-14 10:05:41.315957 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_120dfd6f-cc76-42f9-83fb-954ae00d21c4/
2026-03-14 10:05:41.315974 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_120dfd6f-cc76-42f9-83fb-954ae00d21c4/alertmanager/
2026-03-14 10:05:41.315991 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_120dfd6f-cc76-42f9-83fb-954ae00d21c4/alertmanager/0.log
2026-03-14 10:05:41.316009 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_120dfd6f-cc76-42f9-83fb-954ae00d21c4/config-reloader/
2026-03-14 10:05:41.316026 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_120dfd6f-cc76-42f9-83fb-954ae00d21c4/config-reloader/0.log
2026-03-14 10:05:41.316043 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_120dfd6f-cc76-42f9-83fb-954ae00d21c4/init-config-reloader/
2026-03-14 10:05:41.316060 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_120dfd6f-cc76-42f9-83fb-954ae00d21c4/init-config-reloader/0.log
2026-03-14 10:05:41.316077 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_0b50529a-c274-4664-b0ef-a06045bbfb96/
2026-03-14 10:05:41.316099 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_0b50529a-c274-4664-b0ef-a06045bbfb96/ceilometer-central-agent/
2026-03-14 10:05:41.316117 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_0b50529a-c274-4664-b0ef-a06045bbfb96/ceilometer-central-agent/0.log
2026-03-14 10:05:41.316134 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_0b50529a-c274-4664-b0ef-a06045bbfb96/ceilometer-notification-agent/
2026-03-14 10:05:41.316151 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_0b50529a-c274-4664-b0ef-a06045bbfb96/ceilometer-notification-agent/0.log
2026-03-14 10:05:41.316167 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_0b50529a-c274-4664-b0ef-a06045bbfb96/proxy-httpd/
2026-03-14 10:05:41.316184 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_0b50529a-c274-4664-b0ef-a06045bbfb96/proxy-httpd/0.log
2026-03-14 10:05:41.316201 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_0b50529a-c274-4664-b0ef-a06045bbfb96/sg-core/
2026-03-14 10:05:41.316217 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_0b50529a-c274-4664-b0ef-a06045bbfb96/sg-core/0.log
2026-03-14 10:05:41.316235 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-7f5697c5fb-hqksz_494a65cb-9ba0-448d-944c-7040f8e30bf7/
2026-03-14 10:05:41.316251 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-7f5697c5fb-hqksz_494a65cb-9ba0-448d-944c-7040f8e30bf7/keystone-api/
2026-03-14 10:05:41.316268 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-7f5697c5fb-hqksz_494a65cb-9ba0-448d-944c-7040f8e30bf7/keystone-api/0.log
2026-03-14 10:05:41.316285 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-cron-29558041-khbnb_e95cc8ec-02fb-4344-a08e-ccd9a99f47bd/
2026-03-14 10:05:41.316302 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-cron-29558041-khbnb_e95cc8ec-02fb-4344-a08e-ccd9a99f47bd/keystone-cron/
2026-03-14 10:05:41.316318 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-cron-29558041-khbnb_e95cc8ec-02fb-4344-a08e-ccd9a99f47bd/keystone-cron/0.log
2026-03-14 10:05:41.316335 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_kube-state-metrics-0_4e9ebb9e-ebf4-4700-b5e2-cd76d5bccced/
2026-03-14 10:05:41.316352 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_kube-state-metrics-0_4e9ebb9e-ebf4-4700-b5e2-cd76d5bccced/kube-state-metrics/
2026-03-14 10:05:41.316369 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_kube-state-metrics-0_4e9ebb9e-ebf4-4700-b5e2-cd76d5bccced/kube-state-metrics/0.log
2026-03-14 10:05:41.316386 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_memcached-0_08eba4dc-5f7e-47fe-994d-72af55c68a77/
2026-03-14 10:05:41.316404 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_memcached-0_08eba4dc-5f7e-47fe-994d-72af55c68a77/memcached/
2026-03-14 10:05:41.316421 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_memcached-0_08eba4dc-5f7e-47fe-994d-72af55c68a77/memcached/0.log.gz
2026-03-14 10:05:41.316438 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstack-galera-0_fe7fb9b9-cb1c-47d3-be1e-461c7ea83b43/
2026-03-14 10:05:41.316458 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstack-galera-0_fe7fb9b9-cb1c-47d3-be1e-461c7ea83b43/galera/
2026-03-14 10:05:41.316475 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstack-galera-0_fe7fb9b9-cb1c-47d3-be1e-461c7ea83b43/galera/0.log
2026-03-14 10:05:41.316493 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstack-galera-0_fe7fb9b9-cb1c-47d3-be1e-461c7ea83b43/mysql-bootstrap/
2026-03-14 10:05:41.316509 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstack-galera-0_fe7fb9b9-cb1c-47d3-be1e-461c7ea83b43/mysql-bootstrap/0.log
2026-03-14 10:05:41.316526 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstackclient_c3ef3e95-e2e0-456f-9262-2ae37fb42668/
2026-03-14 10:05:41.316543 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstackclient_c3ef3e95-e2e0-456f-9262-2ae37fb42668/openstackclient/
2026-03-14 10:05:41.316559 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstackclient_c3ef3e95-e2e0-456f-9262-2ae37fb42668/openstackclient/0.log
2026-03-14 10:05:41.316576 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_b4349fd9-0816-49bf-9c0c-c3cbbabe2ff2/
2026-03-14 10:05:41.316594 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_b4349fd9-0816-49bf-9c0c-c3cbbabe2ff2/config-reloader/
2026-03-14 10:05:41.316611 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_b4349fd9-0816-49bf-9c0c-c3cbbabe2ff2/config-reloader/0.log
2026-03-14 10:05:41.316627 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_b4349fd9-0816-49bf-9c0c-c3cbbabe2ff2/init-config-reloader/
2026-03-14 10:05:41.316645 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_b4349fd9-0816-49bf-9c0c-c3cbbabe2ff2/init-config-reloader/0.log
2026-03-14 10:05:41.316662 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_b4349fd9-0816-49bf-9c0c-c3cbbabe2ff2/prometheus/
2026-03-14 10:05:41.316698 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_b4349fd9-0816-49bf-9c0c-c3cbbabe2ff2/prometheus/0.log
2026-03-14 10:05:41.316717 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_b4349fd9-0816-49bf-9c0c-c3cbbabe2ff2/thanos-sidecar/
2026-03-14 10:05:41.316735 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_b4349fd9-0816-49bf-9c0c-c3cbbabe2ff2/thanos-sidecar/0.log
2026-03-14 10:05:41.316752 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-notifications-server-0_eaec015a-59c1-492c-9487-003e222f49fc/
2026-03-14 10:05:41.316769 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-notifications-server-0_eaec015a-59c1-492c-9487-003e222f49fc/rabbitmq/
2026-03-14 10:05:41.316785 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-notifications-server-0_eaec015a-59c1-492c-9487-003e222f49fc/rabbitmq/0.log
2026-03-14 10:05:41.316802 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-notifications-server-0_eaec015a-59c1-492c-9487-003e222f49fc/setup-container/
2026-03-14 10:05:41.316824 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-notifications-server-0_eaec015a-59c1-492c-9487-003e222f49fc/setup-container/0.log
2026-03-14 10:05:41.316840 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-server-0_ced48f61-fae8-48aa-819f-c26f7296dac5/
2026-03-14 10:05:41.316858 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-server-0_ced48f61-fae8-48aa-819f-c26f7296dac5/rabbitmq/
2026-03-14 10:05:41.316874 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-server-0_ced48f61-fae8-48aa-819f-c26f7296dac5/rabbitmq/0.log
2026-03-14 10:05:41.316892 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-server-0_ced48f61-fae8-48aa-819f-c26f7296dac5/setup-container/
2026-03-14 10:05:41.316909 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-server-0_ced48f61-fae8-48aa-819f-c26f7296dac5/setup-container/0.log
2026-03-14 10:05:41.316927 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/
2026-03-14 10:05:41.316945 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/event-filter.html
2026-03-14 10:05:41.316962 | controller | cL+++++++++ ci-framework-data/logs/openstack-must-gather/latest -> quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3
2026-03-14 10:05:41.316980 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/must-gather.logs
2026-03-14 10:05:41.316997 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/timestamp
2026-03-14 10:05:41.317014 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/
2026-03-14 10:05:41.317032 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/all-containers-cpu-top.log
2026-03-14 10:05:41.317049 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/gather.logs
2026-03-14 10:05:41.317066 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/must-gather.tar.xz
2026-03-14 10:05:41.317082 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/packagemanifests.gz
2026-03-14 10:05:41.317099 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/run.log
2026-03-14 10:05:41.317115 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/version
2026-03-14 10:05:41.317132 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/
2026-03-14 10:05:41.317149 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.barbican.openstack.org.yaml
2026-03-14 10:05:41.317170 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.baremetal.openstack.org.yaml
2026-03-14 10:05:41.317188 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.cinder.openstack.org.yaml
2026-03-14 10:05:41.317209 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.client.openstack.org.yaml
2026-03-14 10:05:41.317228 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.core.openstack.org.yaml
2026-03-14 10:05:41.317246 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.dataplane.openstack.org.yaml
2026-03-14 10:05:41.317263 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.designate.openstack.org.yaml
2026-03-14 10:05:41.317280 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.glance.openstack.org.yaml
2026-03-14 10:05:41.317297 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.heat.openstack.org.yaml
2026-03-14 10:05:41.317314 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.horizon.openstack.org.yaml
2026-03-14 10:05:41.317331 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.instanceha.openstack.org.yaml
2026-03-14 10:05:41.317348 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.ironic.openstack.org.yaml
2026-03-14 10:05:41.317365 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.keystone.openstack.org.yaml
2026-03-14 10:05:41.317382 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.manila.openstack.org.yaml
2026-03-14 10:05:41.317398 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.mariadb.openstack.org.yaml
2026-03-14 10:05:41.317415 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.memcached.openstack.org.yaml
2026-03-14 10:05:41.317437 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.network.openstack.org.yaml
2026-03-14 10:05:41.317454 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.neutron.openstack.org.yaml
2026-03-14 10:05:41.317472 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.nova.openstack.org.yaml
2026-03-14 10:05:41.317489 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.octavia.openstack.org.yaml
2026-03-14 10:05:41.317506 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.operator.openstack.org.yaml
2026-03-14 10:05:41.317523 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.ovn.openstack.org.yaml
2026-03-14 10:05:41.317539 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.placement.openstack.org.yaml
2026-03-14 10:05:41.317557 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.rabbitmq.com.yaml
2026-03-14 10:05:41.317574 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.rabbitmq.openstack.org.yaml
2026-03-14 10:05:41.317592 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.redis.openstack.org.yaml
2026-03-14 10:05:41.317609 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.swift.openstack.org.yaml
2026-03-14 10:05:41.317626 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.telemetry.openstack.org.yaml
2026-03-14 10:05:41.317642 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.test.openstack.org.yaml
2026-03-14 10:05:41.317659 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.topology.openstack.org.yaml
2026-03-14 10:05:41.317696 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.watcher.openstack.org.yaml
2026-03-14 10:05:41.317723 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/
2026-03-14 10:05:41.317742 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ansibletests.test.openstack.org.yaml
2026-03-14 10:05:41.317759 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/autoscalings.telemetry.openstack.org.yaml
2026-03-14 10:05:41.317776 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/barbicanapis.barbican.openstack.org.yaml
2026-03-14 10:05:41.317793 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/barbicankeystonelisteners.barbican.openstack.org.yaml
2026-03-14 10:05:41.317810 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/barbicans.barbican.openstack.org.yaml
2026-03-14 10:05:41.317827 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/barbicanworkers.barbican.openstack.org.yaml
2026-03-14 10:05:41.317844 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/bgpconfigurations.network.openstack.org.yaml
2026-03-14 10:05:41.317861 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ceilometers.telemetry.openstack.org.yaml
2026-03-14 10:05:41.317878 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cinderapis.cinder.openstack.org.yaml
2026-03-14 10:05:41.317895 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cinderbackups.cinder.openstack.org.yaml
2026-03-14 10:05:41.317912 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cinders.cinder.openstack.org.yaml
2026-03-14 10:05:41.317929 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cinderschedulers.cinder.openstack.org.yaml
2026-03-14 10:05:41.317946 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cindervolumes.cinder.openstack.org.yaml
2026-03-14 10:05:41.317962 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cloudkitties.telemetry.openstack.org.yaml
2026-03-14 10:05:41.317979 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cloudkittyapis.telemetry.openstack.org.yaml
2026-03-14 10:05:41.318000 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cloudkittyprocs.telemetry.openstack.org.yaml
2026-03-14 10:05:41.318017 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designateapis.designate.openstack.org.yaml
2026-03-14 10:05:41.318034 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designatebackendbind9s.designate.openstack.org.yaml
2026-03-14 10:05:41.318051 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designatecentrals.designate.openstack.org.yaml
2026-03-14 10:05:41.318068 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designatemdnses.designate.openstack.org.yaml
2026-03-14 10:05:41.318085 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designateproducers.designate.openstack.org.yaml
2026-03-14 10:05:41.318102 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designates.designate.openstack.org.yaml
2026-03-14 10:05:41.318118 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designateunbounds.designate.openstack.org.yaml
2026-03-14 10:05:41.318142 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designateworkers.designate.openstack.org.yaml
2026-03-14 10:05:41.318165 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/dnsdata.network.openstack.org.yaml
2026-03-14 10:05:41.318187 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/dnsmasqs.network.openstack.org.yaml
2026-03-14 10:05:41.318208 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/galerabackups.mariadb.openstack.org.yaml
2026-03-14 10:05:41.318230 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/galerarestores.mariadb.openstack.org.yaml
2026-03-14 10:05:41.318252 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/galeras.mariadb.openstack.org.yaml
2026-03-14 10:05:41.318273 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/glanceapis.glance.openstack.org.yaml
2026-03-14 10:05:41.318298 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/glances.glance.openstack.org.yaml
2026-03-14 10:05:41.318322 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/heatapis.heat.openstack.org.yaml
2026-03-14 10:05:41.318344 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/heatcfnapis.heat.openstack.org.yaml
2026-03-14 10:05:41.318365 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/heatengines.heat.openstack.org.yaml
2026-03-14 10:05:41.318387 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/heats.heat.openstack.org.yaml
2026-03-14 10:05:41.318408 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/horizons.horizon.openstack.org.yaml
2026-03-14 10:05:41.318430 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/horizontests.test.openstack.org.yaml
2026-03-14 10:05:41.318452 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/instancehas.instanceha.openstack.org.yaml
2026-03-14 10:05:41.318474 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ipsets.network.openstack.org.yaml
2026-03-14 10:05:41.318496 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ironicapis.ironic.openstack.org.yaml
2026-03-14 10:05:41.318518 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ironicconductors.ironic.openstack.org.yaml
2026-03-14 10:05:41.318540 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ironicinspectors.ironic.openstack.org.yaml
2026-03-14 10:05:41.318561 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ironicneutronagents.ironic.openstack.org.yaml
2026-03-14 10:05:41.318583 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ironics.ironic.openstack.org.yaml
2026-03-14 10:05:41.318605 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/keystoneapis.keystone.openstack.org.yaml
2026-03-14 10:05:41.318630 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/keystoneapplicationcredentials.keystone.openstack.org.yaml
2026-03-14 10:05:41.318653 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/keystoneendpoints.keystone.openstack.org.yaml
2026-03-14 10:05:41.318688 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/keystoneservices.keystone.openstack.org.yaml
2026-03-14 10:05:41.318778 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/loggings.telemetry.openstack.org.yaml
2026-03-14 10:05:41.318802 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/manilaapis.manila.openstack.org.yaml
2026-03-14 10:05:41.318820 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/manilas.manila.openstack.org.yaml
2026-03-14 10:05:41.318837 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/manilaschedulers.manila.openstack.org.yaml
2026-03-14 10:05:41.318854 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/manilashares.manila.openstack.org.yaml
2026-03-14 10:05:41.318871 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/mariadbaccounts.mariadb.openstack.org.yaml
2026-03-14 10:05:41.318888 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/mariadbdatabases.mariadb.openstack.org.yaml
2026-03-14 10:05:41.318905 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/memcacheds.memcached.openstack.org.yaml
2026-03-14 10:05:41.318922 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/metricstorages.telemetry.openstack.org.yaml
2026-03-14 10:05:41.318938 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/netconfigs.network.openstack.org.yaml
2026-03-14 10:05:41.318955 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/neutronapis.neutron.openstack.org.yaml
2026-03-14 10:05:41.318972 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/nova.nova.openstack.org.yaml
2026-03-14 10:05:41.318993 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novaapis.nova.openstack.org.yaml
2026-03-14 10:05:41.319010 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novacells.nova.openstack.org.yaml
2026-03-14 10:05:41.319026 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novacomputes.nova.openstack.org.yaml
2026-03-14 10:05:41.319043 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novaconductors.nova.openstack.org.yaml
2026-03-14 10:05:41.319074 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novametadata.nova.openstack.org.yaml
2026-03-14 10:05:41.319094 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novanovncproxies.nova.openstack.org.yaml
2026-03-14 10:05:41.319112 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novaschedulers.nova.openstack.org.yaml
2026-03-14 10:05:41.319133 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/octaviaamphoracontrollers.octavia.openstack.org.yaml
2026-03-14 10:05:41.319151 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/octaviaapis.octavia.openstack.org.yaml
2026-03-14 10:05:41.319169 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/octaviarsyslogs.octavia.openstack.org.yaml
2026-03-14 10:05:41.319186 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/octavias.octavia.openstack.org.yaml
2026-03-14 10:05:41.319204 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackbaremetalsets.baremetal.openstack.org.yaml
2026-03-14 10:05:41.319222 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackclients.client.openstack.org.yaml
2026-03-14 10:05:41.319239 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackcontrolplanes.core.openstack.org.yaml
2026-03-14 10:05:41.319257 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackdataplanedeployments.dataplane.openstack.org.yaml
2026-03-14 10:05:41.319279 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackdataplanenodesets.dataplane.openstack.org.yaml
2026-03-14 10:05:41.319298 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackdataplaneservices.dataplane.openstack.org.yaml
2026-03-14 10:05:41.319316 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackprovisionservers.baremetal.openstack.org.yaml
2026-03-14 10:05:41.319332 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstacks.operator.openstack.org.yaml
2026-03-14 10:05:41.319350 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackversions.core.openstack.org.yaml
2026-03-14 10:05:41.319367 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ovncontrollers.ovn.openstack.org.yaml
2026-03-14 10:05:41.319384 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ovndbclusters.ovn.openstack.org.yaml
2026-03-14 10:05:41.319401 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ovnnorthds.ovn.openstack.org.yaml
2026-03-14 10:05:41.319418 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/placementapis.placement.openstack.org.yaml
2026-03-14 10:05:41.319435 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/rabbitmqclusters.rabbitmq.com.yaml
2026-03-14 10:05:41.319452 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/rabbitmqpolicies.rabbitmq.openstack.org.yaml
2026-03-14 10:05:41.319469 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/rabbitmqs.rabbitmq.openstack.org.yaml
2026-03-14 10:05:41.319486 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/rabbitmqusers.rabbitmq.openstack.org.yaml
2026-03-14 10:05:41.319502 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/rabbitmqvhosts.rabbitmq.openstack.org.yaml
2026-03-14 10:05:41.319519 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/redises.redis.openstack.org.yaml
2026-03-14 10:05:41.319540 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/reservations.network.openstack.org.yaml
2026-03-14 10:05:41.319558 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/swiftproxies.swift.openstack.org.yaml
2026-03-14 10:05:41.319575 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/swiftrings.swift.openstack.org.yaml
2026-03-14 10:05:41.319591 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/swifts.swift.openstack.org.yaml
2026-03-14 10:05:41.319608 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/swiftstorages.swift.openstack.org.yaml
2026-03-14 10:05:41.319625 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/telemetries.telemetry.openstack.org.yaml
2026-03-14 10:05:41.319642 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/tempests.test.openstack.org.yaml
2026-03-14 10:05:41.319658 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/tobikoes.test.openstack.org.yaml
2026-03-14 10:05:41.319696 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/topologies.topology.openstack.org.yaml
2026-03-14 10:05:41.319719 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/transporturls.rabbitmq.openstack.org.yaml
2026-03-14 10:05:41.319737 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/watcherapis.watcher.openstack.org.yaml
2026-03-14 10:05:41.319754 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/watcherappliers.watcher.openstack.org.yaml
2026-03-14 10:05:41.319771 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/watcherdecisionengines.watcher.openstack.org.yaml
2026-03-14 10:05:41.319788 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/watchers.watcher.openstack.org.yaml
2026-03-14 10:05:41.319805 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/csv/
2026-03-14 10:05:41.319822 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/csv/cluster-observability-operator.v1.3.1
2026-03-14 10:05:41.319844 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/csv/kubernetes-nmstate-operator.4.18.0-202603041813
2026-03-14 10:05:41.319863 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/csv/metallb-operator.v4.18.0-202603040208
2026-03-14 10:05:41.319880 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/csv/openstack-operator.v0.6.0
2026-03-14 10:05:41.319897 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/csv/watcher-operator.v0.0.1
2026-03-14 10:05:41.319914 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/ctlplane/
2026-03-14 10:05:41.319931 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/ctlplane/endpoints
2026-03-14 10:05:41.319948 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/ctlplane/services
2026-03-14 10:05:41.319965 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/ctlplane/ovn/
2026-03-14 10:05:41.319982 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/ctlplane/rabbitmq/
2026-03-14 10:05:41.319999 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/dbs/
2026-03-14 10:05:41.320016 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/dbs/watcher-kuttl-default/
2026-03-14 10:05:41.320032 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/dbs/watcher-kuttl-default/openstack-galera-0-openstack_databases.sql.gz
2026-03-14 10:05:41.320050 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/
2026-03-14 10:05:41.320066 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/
2026-03-14 10:05:41.320083 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/all_resources.log
2026-03-14 10:05:41.320099 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/events.log
2026-03-14 10:05:41.320124 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/nad.log
2026-03-14 10:05:41.320141 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pvc.log
2026-03-14 10:05:41.320158 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/buildconfig/
2026-03-14 10:05:41.320175 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/cert/
2026-03-14 10:05:41.320192 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/certificaterequests/
2026-03-14 10:05:41.320209 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/configmaps/
2026-03-14 10:05:41.320226 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/cronjobs/
2026-03-14 10:05:41.320243 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/daemonset/
2026-03-14 10:05:41.320260 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/deployments/
2026-03-14 10:05:41.320276 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/deployments/cert-manager-cainjector.yaml
2026-03-14 10:05:41.320292 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/deployments/cert-manager-webhook.yaml
2026-03-14 10:05:41.320309 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/deployments/cert-manager.yaml
2026-03-14 10:05:41.320326 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/imagestream/
2026-03-14 10:05:41.320343 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/installplans/
2026-03-14 10:05:41.320359 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/issuer/
2026-03-14 10:05:41.320380 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/jobs/
2026-03-14 10:05:41.320397 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/poddisruptionbudgets/
2026-03-14 10:05:41.320414 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/
2026-03-14 10:05:41.320431 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-545d4d4674-d6lvd/
2026-03-14 10:05:41.320447 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-545d4d4674-d6lvd/cert-manager-545d4d4674-d6lvd-describe
2026-03-14 10:05:41.320465 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-545d4d4674-d6lvd/logs/
2026-03-14 10:05:41.320482 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-545d4d4674-d6lvd/logs/cert-manager-controller.log
2026-03-14 10:05:41.320500 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-cainjector-5545bd876-r4hhn/
2026-03-14 10:05:41.320517 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-cainjector-5545bd876-r4hhn/cert-manager-cainjector-5545bd876-r4hhn-describe
2026-03-14 10:05:41.320535 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-cainjector-5545bd876-r4hhn/logs/
2026-03-14 10:05:41.320552 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-cainjector-5545bd876-r4hhn/logs/cert-manager-cainjector.log
2026-03-14 10:05:41.320569 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-webhook-6888856db4-2rbp4/
2026-03-14 10:05:41.320586 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-webhook-6888856db4-2rbp4/cert-manager-webhook-6888856db4-2rbp4-describe
2026-03-14 10:05:41.320603 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-webhook-6888856db4-2rbp4/logs/
2026-03-14 10:05:41.320624 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-webhook-6888856db4-2rbp4/logs/cert-manager-webhook.log
2026-03-14 10:05:41.320641 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/replicaset/
2026-03-14 10:05:41.320658 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/replicaset/cert-manager-545d4d4674.yaml
2026-03-14 10:05:41.320690 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/replicaset/cert-manager-cainjector-5545bd876.yaml
2026-03-14 10:05:41.320713 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/replicaset/cert-manager-webhook-6888856db4.yaml
2026-03-14 10:05:41.320732 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/routes/
2026-03-14 10:05:41.320749 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/services/
2026-03-14 10:05:41.320766 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/services/cert-manager-cainjector.yaml
2026-03-14 10:05:41.320783 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/services/cert-manager-webhook.yaml
2026-03-14 10:05:41.320800 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/services/cert-manager.yaml
2026-03-14 10:05:41.320817 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/statefulset/
2026-03-14 10:05:41.320834 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/subscriptions/
2026-03-14 10:05:41.320851 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/
2026-03-14 10:05:41.320868 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/all_resources.log
2026-03-14 10:05:41.320885 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/events.log
2026-03-14 10:05:41.320906 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/nad.log
2026-03-14 10:05:41.320924 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pvc.log
2026-03-14 10:05:41.320940 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/buildconfig/
2026-03-14 10:05:41.320958 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/cert/
2026-03-14 10:05:41.320974 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/certificaterequests/
2026-03-14 10:05:41.320991 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/configmaps/
2026-03-14 10:05:41.321008 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/cronjobs/
2026-03-14 10:05:41.321023 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/
2026-03-14 10:05:41.321040 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/ipaddresspools.metallb.io/
2026-03-14 10:05:41.321058 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/ipaddresspools.metallb.io/ctlplane.yaml
2026-03-14 10:05:41.321076 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/ipaddresspools.metallb.io/designateext.yaml
2026-03-14 10:05:41.321093 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/ipaddresspools.metallb.io/internalapi.yaml
2026-03-14 10:05:41.321110 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/ipaddresspools.metallb.io/storage.yaml
2026-03-14 10:05:41.321127 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/ipaddresspools.metallb.io/tenant.yaml
2026-03-14 10:05:41.321144 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/l2advertisements.metallb.io/
2026-03-14 10:05:41.321165 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/l2advertisements.metallb.io/ctlplane.yaml
2026-03-14 10:05:41.321183 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/l2advertisements.metallb.io/designateext.yaml
2026-03-14 10:05:41.321201 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/l2advertisements.metallb.io/internalapi.yaml
2026-03-14 10:05:41.321222 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/l2advertisements.metallb.io/storage.yaml
2026-03-14 10:05:41.321242 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/l2advertisements.metallb.io/tenant.yaml
2026-03-14 10:05:41.321263 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/metallbs.metallb.io/
2026-03-14 10:05:41.321283 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/metallbs.metallb.io/metallb.yaml
2026-03-14 10:05:41.321305 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/daemonset/
2026-03-14 10:05:41.321322 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/daemonset/frr-k8s.yaml
2026-03-14 10:05:41.321339 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/daemonset/speaker.yaml
2026-03-14 10:05:41.321356 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/deployments/
2026-03-14 10:05:41.321373 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/deployments/controller.yaml
2026-03-14 10:05:41.321389 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/deployments/frr-k8s-webhook-server.yaml
2026-03-14 10:05:41.321406 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/deployments/metallb-operator-controller-manager.yaml
2026-03-14 10:05:41.321427 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/deployments/metallb-operator-webhook-server.yaml
2026-03-14 10:05:41.321444 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/imagestream/
2026-03-14 10:05:41.321461 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/installplans/
2026-03-14 10:05:41.321480 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/installplans/install-z4cfd
2026-03-14 10:05:41.321501 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/issuer/
2026-03-14 10:05:41.321522 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/jobs/
2026-03-14 10:05:41.321543 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/poddisruptionbudgets/
2026-03-14 10:05:41.321561 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/
2026-03-14 10:05:41.321644 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/controller-7bb4cc7c98-8v9j7/
2026-03-14 10:05:41.321673 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/controller-7bb4cc7c98-8v9j7/controller-7bb4cc7c98-8v9j7-describe
2026-03-14 10:05:41.321710 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/controller-7bb4cc7c98-8v9j7/logs/
2026-03-14 10:05:41.321729 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/controller-7bb4cc7c98-8v9j7/logs/controller.log.gz
2026-03-14 10:05:41.321747 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/controller-7bb4cc7c98-8v9j7/logs/kube-rbac-proxy.log
2026-03-14 10:05:41.321765 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-2llz5/
2026-03-14 10:05:41.321782 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-2llz5/frr-k8s-2llz5-describe
2026-03-14 10:05:41.321803 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-2llz5/logs/
2026-03-14 10:05:41.321821 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-2llz5/logs/controller.log
2026-03-14 10:05:41.321838 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-2llz5/logs/cp-frr-files.log
2026-03-14 10:05:41.321855 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-2llz5/logs/cp-metrics.log
2026-03-14 10:05:41.321871 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-2llz5/logs/cp-reloader.log
2026-03-14 10:05:41.321887 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-2llz5/logs/frr-metrics.log
2026-03-14 10:05:41.321904 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-2llz5/logs/frr.log.gz
2026-03-14 10:05:41.321922 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-2llz5/logs/kube-rbac-proxy-frr.log
2026-03-14 10:05:41.321939 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-2llz5/logs/kube-rbac-proxy.log
2026-03-14 10:05:41.321956 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-2llz5/logs/reloader.log
2026-03-14 10:05:41.321973 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-webhook-server-bcc4b6f68-nbb5f/
2026-03-14 10:05:41.321989 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-webhook-server-bcc4b6f68-nbb5f/frr-k8s-webhook-server-bcc4b6f68-nbb5f-describe
2026-03-14 10:05:41.322007 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-webhook-server-bcc4b6f68-nbb5f/logs/
2026-03-14 10:05:41.322024 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-webhook-server-bcc4b6f68-nbb5f/logs/frr-k8s-webhook-server.log
2026-03-14 10:05:41.322045 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-controller-manager-6c5855cb99-gcpx9/
2026-03-14 10:05:41.322063 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-controller-manager-6c5855cb99-gcpx9/metallb-operator-controller-manager-6c5855cb99-gcpx9-describe
2026-03-14 10:05:41.322081 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-controller-manager-6c5855cb99-gcpx9/logs/
2026-03-14 10:05:41.322099 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-controller-manager-6c5855cb99-gcpx9/logs/manager.log
2026-03-14 10:05:41.322118 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-webhook-server-76b978c99b-flxj6/
2026-03-14 10:05:41.322135 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-webhook-server-76b978c99b-flxj6/metallb-operator-webhook-server-76b978c99b-flxj6-describe
2026-03-14 10:05:41.322152 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-webhook-server-76b978c99b-flxj6/logs/
2026-03-14 10:05:41.322170 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-webhook-server-76b978c99b-flxj6/logs/webhook-server.log
2026-03-14 10:05:41.322188 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/speaker-bv84h/
2026-03-14 10:05:41.322206 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/speaker-bv84h/speaker-bv84h-describe
2026-03-14 10:05:41.322222 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/speaker-bv84h/logs/
2026-03-14 10:05:41.322239 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/speaker-bv84h/logs/kube-rbac-proxy.log
2026-03-14 10:05:41.322257 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/speaker-bv84h/logs/speaker.log.gz
2026-03-14 10:05:41.322278 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/replicaset/
2026-03-14 10:05:41.322296 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/replicaset/controller-7bb4cc7c98.yaml
2026-03-14 10:05:41.322313 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/replicaset/frr-k8s-webhook-server-bcc4b6f68.yaml
2026-03-14 10:05:41.322331 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/replicaset/metallb-operator-controller-manager-6c5855cb99.yaml
2026-03-14 10:05:41.322348 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/replicaset/metallb-operator-webhook-server-76b978c99b.yaml
2026-03-14 10:05:41.322365 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/routes/
2026-03-14 10:05:41.322383 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/
2026-03-14 10:05:41.322400 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/controller-monitor-service.yaml
2026-03-14 10:05:41.322419 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/frr-k8s-monitor-service.yaml
2026-03-14 10:05:41.322437 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/frr-k8s-webhook-service.yaml
2026-03-14 10:05:41.322455 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/metallb-operator-controller-manager-service.yaml
2026-03-14 10:05:41.322472 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/metallb-operator-webhook-server-service.yaml
2026-03-14 10:05:41.322489 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/metallb-operator-webhook-service.yaml
2026-03-14 10:05:41.322507 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/speaker-monitor-service.yaml
2026-03-14 10:05:41.322528 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/webhook-service.yaml
2026-03-14 10:05:41.322547 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/statefulset/
2026-03-14 10:05:41.322564 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/subscriptions/
2026-03-14 10:05:41.322581 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/subscriptions/metallb-operator-sub
2026-03-14 10:05:41.322599 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/
2026-03-14 10:05:41.322615 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/all_resources.log
2026-03-14 10:05:41.322633 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/events.log
2026-03-14 10:05:41.322650 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/nad.log
2026-03-14 10:05:41.322668 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pvc.log
2026-03-14 10:05:41.322704 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/buildconfig/
2026-03-14 10:05:41.322722 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/cert/
2026-03-14 10:05:41.322739 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/certificaterequests/
2026-03-14 10:05:41.322756 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/configmaps/
2026-03-14 10:05:41.322773 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/cronjobs/
2026-03-14 10:05:41.322790 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/daemonset/
2026-03-14 10:05:41.322811 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/deployments/
2026-03-14 10:05:41.322828 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml
2026-03-14 10:05:41.322846 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/deployments/machine-api-operator.yaml
2026-03-14 10:05:41.322863 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/imagestream/
2026-03-14 10:05:41.322880 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/installplans/
2026-03-14 10:05:41.322897 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/issuer/
2026-03-14 10:05:41.322914 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/jobs/
2026-03-14 10:05:41.322932 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/poddisruptionbudgets/
2026-03-14 10:05:41.322948 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/
2026-03-14 10:05:41.322965 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-78cbb6b69f-cxr6c/
2026-03-14 10:05:41.322982 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-78cbb6b69f-cxr6c/control-plane-machine-set-operator-78cbb6b69f-cxr6c-describe
2026-03-14 10:05:41.322999 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-78cbb6b69f-cxr6c/logs/
2026-03-14 10:05:41.323017 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-78cbb6b69f-cxr6c/logs/control-plane-machine-set-operator.log
2026-03-14 10:05:41.323034 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/machine-api-operator-5694c8668f-5kqvs/
2026-03-14 10:05:41.323067 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/machine-api-operator-5694c8668f-5kqvs/machine-api-operator-5694c8668f-5kqvs-describe
2026-03-14 10:05:41.323089 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/machine-api-operator-5694c8668f-5kqvs/logs/
2026-03-14 10:05:41.323113 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/machine-api-operator-5694c8668f-5kqvs/logs/kube-rbac-proxy.log
2026-03-14 10:05:41.323134 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/machine-api-operator-5694c8668f-5kqvs/logs/machine-api-operator.log
2026-03-14 10:05:41.323152 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/replicaset/
2026-03-14 10:05:41.323171 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-78cbb6b69f.yaml
2026-03-14 10:05:41.323189 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/replicaset/machine-api-operator-5694c8668f.yaml
2026-03-14 10:05:41.323208 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/routes/
2026-03-14 10:05:41.323226 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/
2026-03-14 10:05:41.323245 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml
2026-03-14 10:05:41.323263 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml
2026-03-14 10:05:41.323282 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/machine-api-controllers.yaml
2026-03-14 10:05:41.323300 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml
2026-03-14 10:05:41.323322 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml
2026-03-14 10:05:41.323339 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/machine-api-operator.yaml
2026-03-14 10:05:41.323357 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/statefulset/
2026-03-14 10:05:41.323374 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/subscriptions/
2026-03-14 10:05:41.323392 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/
2026-03-14 10:05:41.323410 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/all_resources.log
2026-03-14 10:05:41.323427 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/events.log
2026-03-14 10:05:41.323444 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/nad.log
2026-03-14 10:05:41.323461 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pvc.log
2026-03-14 10:05:41.323479 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/buildconfig/
2026-03-14 10:05:41.323496 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/cert/
2026-03-14 10:05:41.323513 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/certificaterequests/
2026-03-14 10:05:41.323530 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/configmaps/
2026-03-14 10:05:41.323547 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/cronjobs/
2026-03-14 10:05:41.323563 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/daemonset/
2026-03-14 10:05:41.323584 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/deployments/
2026-03-14 10:05:41.323602 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/deployments/marketplace-operator.yaml
2026-03-14 10:05:41.323618 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/imagestream/
2026-03-14 10:05:41.323635 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/installplans/
2026-03-14 10:05:41.323653 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/issuer/
2026-03-14 10:05:41.323670 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/jobs/
2026-03-14 10:05:41.323705 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/jobs/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874832f3.yaml
2026-03-14 10:05:41.323724 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/jobs/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c166a6a.yaml
2026-03-14 10:05:41.323742 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/jobs/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e56eb0c.yaml
2026-03-14 10:05:41.323760 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/jobs/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f081954b.yaml
2026-03-14 10:05:41.323777 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/poddisruptionbudgets/
2026-03-14 10:05:41.323795 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/
2026-03-14 10:05:41.323813 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874cs2f2/
2026-03-14 10:05:41.323830 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874cs2f2/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874cs2f2-describe
2026-03-14 10:05:41.323856 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874cs2f2/logs/
2026-03-14 10:05:41.323875 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874cs2f2/logs/extract.log
2026-03-14 10:05:41.323893 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874cs2f2/logs/pull.log
2026-03-14 10:05:41.323911 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874cs2f2/logs/util.log
2026-03-14 10:05:41.323928 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c14m429/
2026-03-14 10:05:41.323946 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c14m429/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c14m429-describe
2026-03-14 10:05:41.323964 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c14m429/logs/
2026-03-14 10:05:41.323982 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c14m429/logs/extract.log
2026-03-14 10:05:41.324000 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c14m429/logs/pull.log
2026-03-14 10:05:41.324018 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c14m429/logs/util.log
2026-03-14 10:05:41.324035 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5qhxpx/
2026-03-14 10:05:41.324053 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5qhxpx/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5qhxpx-describe
2026-03-14 10:05:41.324077 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5qhxpx/logs/
2026-03-14 10:05:41.324095 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5qhxpx/logs/extract.log
2026-03-14 10:05:41.324112 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5qhxpx/logs/pull.log
2026-03-14 10:05:41.324129 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5qhxpx/logs/util.log
2026-03-14 10:05:41.324145 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f0857cj7/
2026-03-14 10:05:41.324162 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f0857cj7/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f0857cj7-describe
2026-03-14 10:05:41.324179 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f0857cj7/logs/
2026-03-14 10:05:41.324196 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f0857cj7/logs/extract.log
2026-03-14 10:05:41.324214 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f0857cj7/logs/pull.log
2026-03-14 10:05:41.324231 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f0857cj7/logs/util.log
2026-03-14 10:05:41.324249 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/certified-operators-tt7cz/
2026-03-14 10:05:41.324266 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/certified-operators-tt7cz/certified-operators-tt7cz-describe
2026-03-14 10:05:41.324287 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/certified-operators-tt7cz/logs/
2026-03-14 10:05:41.324305 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/certified-operators-tt7cz/logs/extract-content.log
2026-03-14 10:05:41.324323 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/certified-operators-tt7cz/logs/extract-utilities.log
2026-03-14 10:05:41.324341 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/certified-operators-tt7cz/logs/registry-server.log.gz
2026-03-14 10:05:41.324358 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/community-operators-llm5h/
2026-03-14 10:05:41.324376 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/community-operators-llm5h/community-operators-llm5h-describe
2026-03-14 10:05:41.324418 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/community-operators-llm5h/logs/
2026-03-14 10:05:41.324444 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/community-operators-llm5h/logs/extract-content.log
2026-03-14 10:05:41.324462 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/community-operators-llm5h/logs/extract-utilities.log
2026-03-14 10:05:41.324479 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/community-operators-llm5h/logs/registry-server.log.gz
2026-03-14 10:05:41.324496 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/marketplace-operator-79b997595-565tx/
2026-03-14 10:05:41.324513 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/marketplace-operator-79b997595-565tx/marketplace-operator-79b997595-565tx-describe
2026-03-14 10:05:41.324530 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/marketplace-operator-79b997595-565tx/logs/
2026-03-14 10:05:41.324552 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/marketplace-operator-79b997595-565tx/logs/marketplace-operator.log
2026-03-14 10:05:41.324570 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-marketplace-tfbnl/
2026-03-14 10:05:41.324588 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-marketplace-tfbnl/redhat-marketplace-tfbnl-describe
2026-03-14 10:05:41.324606 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-marketplace-tfbnl/logs/
2026-03-14 10:05:41.324623 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-marketplace-tfbnl/logs/extract-content.log
2026-03-14 10:05:41.324641 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-marketplace-tfbnl/logs/extract-utilities.log
2026-03-14 10:05:41.324658 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-marketplace-tfbnl/logs/registry-server.log
2026-03-14 10:05:41.324689 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-operators-vx4kc/
2026-03-14 10:05:41.324712 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-operators-vx4kc/redhat-operators-vx4kc-describe
2026-03-14 10:05:41.324731 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-operators-vx4kc/logs/
2026-03-14 10:05:41.324749 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-operators-vx4kc/logs/extract-content.log
2026-03-14 10:05:41.324766 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-operators-vx4kc/logs/extract-utilities.log
2026-03-14 10:05:41.324783 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-operators-vx4kc/logs/registry-server.log.gz
2026-03-14 10:05:41.324805 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/replicaset/
2026-03-14 10:05:41.324823 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/replicaset/marketplace-operator-79b997595.yaml
2026-03-14 10:05:41.324842 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/routes/
2026-03-14 10:05:41.324859 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/services/
2026-03-14 10:05:41.324877 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/services/certified-operators.yaml
2026-03-14 10:05:41.324894 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/services/community-operators.yaml
2026-03-14 10:05:41.324911 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/services/marketplace-operator-metrics.yaml
2026-03-14 10:05:41.324929 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/services/redhat-marketplace.yaml
2026-03-14 10:05:41.324946 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/services/redhat-operators.yaml
2026-03-14 10:05:41.324963 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/statefulset/
2026-03-14 10:05:41.324981 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/subscriptions/
2026-03-14 10:05:41.324999 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/
2026-03-14 10:05:41.325016 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/all_resources.log
2026-03-14 10:05:41.325034 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/events.log
2026-03-14 10:05:41.325051 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/nad.log
2026-03-14 10:05:41.325072 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pvc.log
2026-03-14 10:05:41.325091 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/buildconfig/
2026-03-14 10:05:41.325109 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/cert/
2026-03-14 10:05:41.325126 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/certificaterequests/
2026-03-14 10:05:41.325143 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/configmaps/
2026-03-14 10:05:41.325160 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/cronjobs/
2026-03-14 10:05:41.325176 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/daemonset/
2026-03-14 10:05:41.325193 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/daemonset/nmstate-handler.yaml
2026-03-14 10:05:41.325210 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/deployments/
2026-03-14 10:05:41.325228 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/deployments/nmstate-console-plugin.yaml
2026-03-14 10:05:41.325245 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/deployments/nmstate-metrics.yaml
2026-03-14 10:05:41.325262 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/deployments/nmstate-operator.yaml
2026-03-14 10:05:41.325279 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/deployments/nmstate-webhook.yaml
2026-03-14 10:05:41.325297 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/imagestream/
2026-03-14 10:05:41.325315 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/installplans/
2026-03-14 10:05:41.325337 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/installplans/install-zfzh8
2026-03-14 10:05:41.325355 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/issuer/
2026-03-14 10:05:41.325372 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/jobs/
2026-03-14 10:05:41.325389 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/poddisruptionbudgets/
2026-03-14 10:05:41.325406 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/poddisruptionbudgets/nmstate-webhook.yaml
2026-03-14 10:05:41.325423 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/
2026-03-14 10:05:41.325441 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-console-plugin-86f58fcf4-wwth2/
2026-03-14 10:05:41.325458 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-console-plugin-86f58fcf4-wwth2/nmstate-console-plugin-86f58fcf4-wwth2-describe
2026-03-14 10:05:41.325476 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-console-plugin-86f58fcf4-wwth2/logs/
2026-03-14 10:05:41.325493 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-console-plugin-86f58fcf4-wwth2/logs/nmstate-console-plugin.log
2026-03-14 10:05:41.325511 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-handler-fqs5c/
2026-03-14 10:05:41.325528 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-handler-fqs5c/nmstate-handler-fqs5c-describe
2026-03-14 10:05:41.325545 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-handler-fqs5c/logs/
2026-03-14 10:05:41.325561 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-handler-fqs5c/logs/nmstate-handler.log
2026-03-14 10:05:41.325583 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-metrics-9b8c8685d-bcwrg/
2026-03-14 10:05:41.325601 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-metrics-9b8c8685d-bcwrg/nmstate-metrics-9b8c8685d-bcwrg-describe
2026-03-14 10:05:41.325619 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-metrics-9b8c8685d-bcwrg/logs/
2026-03-14 10:05:41.325637 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-metrics-9b8c8685d-bcwrg/logs/kube-rbac-proxy.log
2026-03-14 10:05:41.325654 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-metrics-9b8c8685d-bcwrg/logs/nmstate-metrics.log
2026-03-14 10:05:41.325671 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-operator-796d4cfff4-n7k5d/
2026-03-14 10:05:41.325710 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-operator-796d4cfff4-n7k5d/nmstate-operator-796d4cfff4-n7k5d-describe
2026-03-14 10:05:41.325739 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-operator-796d4cfff4-n7k5d/logs/
2026-03-14 10:05:41.325757 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-operator-796d4cfff4-n7k5d/logs/nmstate-operator.log
2026-03-14 10:05:41.325774 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-webhook-5f558f5558-hcdc6/
2026-03-14 10:05:41.325791 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-webhook-5f558f5558-hcdc6/nmstate-webhook-5f558f5558-hcdc6-describe
2026-03-14 10:05:41.325809 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-webhook-5f558f5558-hcdc6/logs/
2026-03-14 10:05:41.325826 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-webhook-5f558f5558-hcdc6/logs/nmstate-webhook.log
2026-03-14 10:05:41.325849 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/replicaset/
2026-03-14 10:05:41.325867 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/replicaset/nmstate-console-plugin-86f58fcf4.yaml
2026-03-14 10:05:41.325884 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/replicaset/nmstate-metrics-9b8c8685d.yaml
2026-03-14 10:05:41.325900 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/replicaset/nmstate-operator-796d4cfff4.yaml
2026-03-14 10:05:41.325917 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/replicaset/nmstate-webhook-5f558f5558.yaml
2026-03-14 10:05:41.325935 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/routes/
2026-03-14 10:05:41.325952 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/services/
2026-03-14 10:05:41.325968 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/services/nmstate-console-plugin.yaml
2026-03-14 10:05:41.325985 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/services/nmstate-monitor.yaml
2026-03-14 10:05:41.326003 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/services/nmstate-webhook.yaml
2026-03-14 10:05:41.326019 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/statefulset/
2026-03-14 10:05:41.326036 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/subscriptions/
2026-03-14 10:05:41.326053 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/subscriptions/kubernetes-nmstate-operator
2026-03-14 10:05:41.326070 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/
2026-03-14 10:05:41.326092 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/all_resources.log
2026-03-14 10:05:41.326109 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/events.log
2026-03-14 10:05:41.326126 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/nad.log
2026-03-14 10:05:41.326143 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pvc.log
2026-03-14 10:05:41.326160 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/buildconfig/
2026-03-14 10:05:41.326177 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/cert/
2026-03-14 10:05:41.326194 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/certificaterequests/
2026-03-14 10:05:41.326211 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/configmaps/
2026-03-14 10:05:41.326229 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/cronjobs/
2026-03-14 10:05:41.326246 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/daemonset/
2026-03-14 10:05:41.326263 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/deployments/
2026-03-14 10:05:41.326280 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/deployments/obo-prometheus-operator-admission-webhook.yaml
2026-03-14 10:05:41.326296 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/deployments/obo-prometheus-operator.yaml
2026-03-14 10:05:41.326313 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/deployments/observability-operator.yaml
2026-03-14 10:05:41.326330 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/deployments/observability-ui-dashboards.yaml
2026-03-14 10:05:41.326351 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/deployments/perses-operator.yaml
2026-03-14 10:05:41.326369 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/imagestream/
2026-03-14 10:05:41.326386 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/installplans/
2026-03-14 10:05:41.326404 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/installplans/install-bvfdn
2026-03-14 10:05:41.326420 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/issuer/
2026-03-14 10:05:41.326438 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/jobs/
2026-03-14 10:05:41.326455 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/poddisruptionbudgets/
2026-03-14 10:05:41.326472 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/poddisruptionbudgets/obo-prometheus-operator-admission-webhook.yaml
2026-03-14 10:05:41.326488 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/
2026-03-14 10:05:41.326505 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-68bc856cb9-7vsx5/
2026-03-14 10:05:41.326523 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-68bc856cb9-7vsx5/obo-prometheus-operator-68bc856cb9-7vsx5-describe
2026-03-14 10:05:41.326540 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-68bc856cb9-7vsx5/logs/
2026-03-14 10:05:41.326558 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-68bc856cb9-7vsx5/logs/prometheus-operator.log
2026-03-14 10:05:41.326574 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-69fb744b97-5knkk/
2026-03-14 10:05:41.326595 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-69fb744b97-5knkk/obo-prometheus-operator-admission-webhook-69fb744b97-5knkk-describe
2026-03-14 10:05:41.326612 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-69fb744b97-5knkk/logs/
2026-03-14 10:05:41.326629 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-69fb744b97-5knkk/logs/prometheus-operator-admission-webhook.log
2026-03-14 10:05:41.326646 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-69fb744b97-scrd4/
2026-03-14 10:05:41.326663 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-69fb744b97-scrd4/obo-prometheus-operator-admission-webhook-69fb744b97-scrd4-describe
2026-03-14 10:05:41.326699 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-69fb744b97-scrd4/logs/
2026-03-14 10:05:41.326719 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-69fb744b97-scrd4/logs/prometheus-operator-admission-webhook.log
2026-03-14 10:05:41.326737 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-operator-59bdc8b94-7l9bs/
2026-03-14 10:05:41.326754 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-operator-59bdc8b94-7l9bs/observability-operator-59bdc8b94-7l9bs-describe
2026-03-14 10:05:41.326771 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-operator-59bdc8b94-7l9bs/logs/
2026-03-14 10:05:41.326788 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-operator-59bdc8b94-7l9bs/logs/operator.log
2026-03-14 10:05:41.326805 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-ui-dashboards-66cbf594b5-k8bb7/
2026-03-14 10:05:41.326826 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-ui-dashboards-66cbf594b5-k8bb7/observability-ui-dashboards-66cbf594b5-k8bb7-describe
2026-03-14 10:05:41.326843 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-ui-dashboards-66cbf594b5-k8bb7/logs/
2026-03-14 10:05:41.326859 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-ui-dashboards-66cbf594b5-k8bb7/logs/observability-ui-dashboards.log
2026-03-14 10:05:41.326875 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/perses-operator-5bf474d74f-8glvj/
2026-03-14 10:05:41.326892 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/perses-operator-5bf474d74f-8glvj/perses-operator-5bf474d74f-8glvj-describe
2026-03-14 10:05:41.326909 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/perses-operator-5bf474d74f-8glvj/logs/
2026-03-14 10:05:41.326926 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/perses-operator-5bf474d74f-8glvj/logs/perses-operator.log
2026-03-14 10:05:41.326943 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/replicaset/
2026-03-14 10:05:41.326960 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/replicaset/obo-prometheus-operator-68bc856cb9.yaml
2026-03-14 10:05:41.326977 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/replicaset/obo-prometheus-operator-admission-webhook-69fb744b97.yaml
2026-03-14 10:05:41.326994 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/replicaset/observability-operator-59bdc8b94.yaml
2026-03-14 10:05:41.327011 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/replicaset/observability-ui-dashboards-66cbf594b5.yaml
2026-03-14 10:05:41.327028 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/replicaset/perses-operator-5bf474d74f.yaml
2026-03-14 10:05:41.327049 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/routes/
2026-03-14 10:05:41.327083 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/services/
2026-03-14 10:05:41.327103 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/services/obo-prometheus-operator-admission-webhook-service.yaml
2026-03-14 10:05:41.327122 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/services/obo-prometheus-operator-admission-webhook.yaml
2026-03-14 10:05:41.327143 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/services/obo-prometheus-operator.yaml
2026-03-14 10:05:41.327161 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/services/observability-operator.yaml
2026-03-14 10:05:41.327179 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/services/observability-ui-dashboards.yaml
2026-03-14 10:05:41.327198 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/statefulset/
2026-03-14 10:05:41.327216 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/subscriptions/
2026-03-14 10:05:41.327235 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/subscriptions/cluster-observability-operator
2026-03-14 10:05:41.327252 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/subscriptions/observability-operator
2026-03-14 10:05:41.327270 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/
2026-03-14 10:05:41.327288 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/all_resources.log
2026-03-14 10:05:41.327306 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/events.log
2026-03-14 10:05:41.327323 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/nad.log
2026-03-14 10:05:41.327344 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pvc.log
2026-03-14 10:05:41.327361 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/buildconfig/
2026-03-14 10:05:41.327378 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/
2026-03-14 10:05:41.327396 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/barbican-operator-metrics-certs.yaml
2026-03-14 10:05:41.327412 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/cinder-operator-metrics-certs.yaml
2026-03-14 10:05:41.327430 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/designate-operator-metrics-certs.yaml
2026-03-14 10:05:41.327447 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/glance-operator-metrics-certs.yaml
2026-03-14 10:05:41.327464 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/heat-operator-metrics-certs.yaml
2026-03-14 10:05:41.327481 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/horizon-operator-metrics-certs.yaml
2026-03-14 10:05:41.327498 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/infra-operator-metrics-certs.yaml
2026-03-14 10:05:41.327515 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/infra-operator-serving-cert.yaml
2026-03-14 10:05:41.327533 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/ironic-operator-metrics-certs.yaml
2026-03-14 10:05:41.327550 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/keystone-operator-metrics-certs.yaml
2026-03-14 10:05:41.327566 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/manila-operator-metrics-certs.yaml
2026-03-14 10:05:41.327586 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/mariadb-operator-metrics-certs.yaml
2026-03-14 10:05:41.327603 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/neutron-operator-metrics-certs.yaml
2026-03-14 10:05:41.327620 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/nova-operator-metrics-certs.yaml
2026-03-14 10:05:41.327637 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/octavia-operator-metrics-certs.yaml
2026-03-14 10:05:41.327653 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/openstack-baremetal-operator-metrics-certs.yaml
2026-03-14 10:05:41.327691 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/openstack-baremetal-operator-serving-cert.yaml
2026-03-14 10:05:41.327714 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/openstack-operator-metrics-certs.yaml
2026-03-14 10:05:41.327733 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/openstack-operator-serving-cert.yaml
2026-03-14 10:05:41.327750 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/ovn-operator-metrics-certs.yaml
2026-03-14 10:05:41.327767 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/placement-operator-metrics-certs.yaml
2026-03-14 10:05:41.327784 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/swift-operator-metrics-certs.yaml
2026-03-14 10:05:41.327801 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/telemetry-operator-metrics-certs.yaml
2026-03-14 10:05:41.327818 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/test-operator-metrics-certs.yaml
2026-03-14 10:05:41.327835 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/watcher-operator-metrics-certs.yaml
2026-03-14 10:05:41.327860 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/
2026-03-14 10:05:41.327877 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/barbican-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.327895 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/cinder-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.327912 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/designate-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.327929 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/glance-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.327946 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/heat-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.327962 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/horizon-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.327980 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/infra-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.327997 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/infra-operator-serving-cert-1.yaml
2026-03-14 10:05:41.328014 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/ironic-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.328031 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/keystone-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.328048 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/manila-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.328065 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/mariadb-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.328086 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/neutron-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.328104 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/nova-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.328121 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/octavia-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.328138 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/openstack-baremetal-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.328155 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/openstack-baremetal-operator-serving-cert-1.yaml
2026-03-14 10:05:41.328172 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/openstack-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.328190 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/openstack-operator-serving-cert-1.yaml
2026-03-14 10:05:41.328207 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/ovn-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.328223 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/placement-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.328241 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/swift-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.328257 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/telemetry-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.328274 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/test-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.328291 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/watcher-operator-metrics-certs-1.yaml
2026-03-14 10:05:41.328313 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/configmaps/
2026-03-14 10:05:41.328330 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cronjobs/
2026-03-14 10:05:41.328348 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/crs/
2026-03-14 10:05:41.328370 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/crs/openstacks.operator.openstack.org/
2026-03-14 10:05:41.328393 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/crs/openstacks.operator.openstack.org/openstack.yaml
2026-03-14 10:05:41.328414 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/daemonset/
2026-03-14 10:05:41.328433 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/
2026-03-14 10:05:41.328450 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/barbican-operator-controller-manager.yaml
2026-03-14 10:05:41.328467 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/cinder-operator-controller-manager.yaml
2026-03-14 10:05:41.328484 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/designate-operator-controller-manager.yaml
2026-03-14 10:05:41.328501 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/glance-operator-controller-manager.yaml
2026-03-14 10:05:41.328518 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/heat-operator-controller-manager.yaml
2026-03-14 10:05:41.328534 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/horizon-operator-controller-manager.yaml
2026-03-14 10:05:41.328551 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/infra-operator-controller-manager.yaml
2026-03-14 10:05:41.328581 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/ironic-operator-controller-manager.yaml
2026-03-14 10:05:41.328599 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/keystone-operator-controller-manager.yaml
2026-03-14 10:05:41.328615 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/manila-operator-controller-manager.yaml
2026-03-14 10:05:41.328632 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/mariadb-operator-controller-manager.yaml
2026-03-14 10:05:41.328649 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/neutron-operator-controller-manager.yaml
2026-03-14 10:05:41.328673 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/nova-operator-controller-manager.yaml
2026-03-14 10:05:41.328713 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/octavia-operator-controller-manager.yaml
2026-03-14 10:05:41.328732 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/openstack-baremetal-operator-controller-manager.yaml
2026-03-14 10:05:41.328750 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/openstack-operator-controller-init.yaml
2026-03-14 10:05:41.328767 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/openstack-operator-controller-manager.yaml
2026-03-14 10:05:41.328784 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/ovn-operator-controller-manager.yaml
2026-03-14 10:05:41.328802 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/placement-operator-controller-manager.yaml
2026-03-14 10:05:41.328820 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/rabbitmq-cluster-operator-manager.yaml
2026-03-14 10:05:41.328843 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/swift-operator-controller-manager.yaml
2026-03-14 10:05:41.328868 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/telemetry-operator-controller-manager.yaml
2026-03-14 10:05:41.328891 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/test-operator-controller-manager.yaml
2026-03-14 10:05:41.328913 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/watcher-operator-controller-manager.yaml
2026-03-14 10:05:41.328935 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/imagestream/
2026-03-14 10:05:41.328958 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/installplans/
2026-03-14 10:05:41.328980 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/installplans/install-nshs9
2026-03-14 10:05:41.329002 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/installplans/install-phqsn
2026-03-14 10:05:41.329024 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/
2026-03-14 10:05:41.329046 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/barbican-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329069 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/cinder-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329091 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/designate-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329113 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/glance-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329135 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/heat-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329162 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/horizon-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329185 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/infra-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329208 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/ironic-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329229 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/keystone-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329251 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/manila-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329275 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/mariadb-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329303 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/neutron-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329334 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/nova-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329362 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/octavia-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329385 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/openstack-baremetal-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329407 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/openstack-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329429 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/ovn-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329451 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/placement-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329473 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/rabbitmq-cluster-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329499 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/swift-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329522 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/telemetry-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329544 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/test-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329567 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/watcher-operator-selfsigned-issuer.yaml
2026-03-14 10:05:41.329588 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/jobs/
2026-03-14 10:05:41.329611 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/jobs/65270e5b79eb73ea65900207edd02f6e862f5863f7b0ae98982f030d4cb742a.yaml
2026-03-14 10:05:41.329634 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/jobs/b2fc087f8a76dbce8476595fc384d33288943d0097430396a1362c765ea7cc8.yaml
2026-03-14 10:05:41.329656 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/poddisruptionbudgets/
2026-03-14 10:05:41.329741 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/
2026-03-14 10:05:41.329768 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/65270e5b79eb73ea65900207edd02f6e862f5863f7b0ae98982f030d4cpkw4d/
2026-03-14 10:05:41.329787 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/65270e5b79eb73ea65900207edd02f6e862f5863f7b0ae98982f030d4cpkw4d/65270e5b79eb73ea65900207edd02f6e862f5863f7b0ae98982f030d4cpkw4d-describe
2026-03-14 10:05:41.329805 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/65270e5b79eb73ea65900207edd02f6e862f5863f7b0ae98982f030d4cpkw4d/logs/
2026-03-14 10:05:41.329822 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/65270e5b79eb73ea65900207edd02f6e862f5863f7b0ae98982f030d4cpkw4d/logs/extract.log
2026-03-14 10:05:41.329845 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/65270e5b79eb73ea65900207edd02f6e862f5863f7b0ae98982f030d4cpkw4d/logs/pull.log
2026-03-14 10:05:41.329862 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/65270e5b79eb73ea65900207edd02f6e862f5863f7b0ae98982f030d4cpkw4d/logs/util.log
2026-03-14 10:05:41.329879 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/b2fc087f8a76dbce8476595fc384d33288943d0097430396a1362c765eshgz7/
2026-03-14 10:05:41.329897 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/b2fc087f8a76dbce8476595fc384d33288943d0097430396a1362c765eshgz7/b2fc087f8a76dbce8476595fc384d33288943d0097430396a1362c765eshgz7-describe
2026-03-14 10:05:41.329914 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/b2fc087f8a76dbce8476595fc384d33288943d0097430396a1362c765eshgz7/logs/
2026-03-14 10:05:41.329932 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/b2fc087f8a76dbce8476595fc384d33288943d0097430396a1362c765eshgz7/logs/extract.log
2026-03-14 10:05:41.329950 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/b2fc087f8a76dbce8476595fc384d33288943d0097430396a1362c765eshgz7/logs/pull.log
2026-03-14 10:05:41.329967 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/b2fc087f8a76dbce8476595fc384d33288943d0097430396a1362c765eshgz7/logs/util.log
2026-03-14 10:05:41.329984 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/barbican-operator-controller-manager-d47688694-ktr4z/
2026-03-14 10:05:41.330001 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/barbican-operator-controller-manager-d47688694-ktr4z/barbican-operator-controller-manager-d47688694-ktr4z-describe
2026-03-14 10:05:41.330018 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/barbican-operator-controller-manager-d47688694-ktr4z/logs/
2026-03-14 10:05:41.330036 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/barbican-operator-controller-manager-d47688694-ktr4z/logs/manager.log
2026-03-14 10:05:41.330057 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/cinder-operator-controller-manager-984cd4dcf-75kt6/
2026-03-14 10:05:41.330076 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/cinder-operator-controller-manager-984cd4dcf-75kt6/cinder-operator-controller-manager-984cd4dcf-75kt6-describe
2026-03-14 10:05:41.330093 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/cinder-operator-controller-manager-984cd4dcf-75kt6/logs/
2026-03-14 10:05:41.330110 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/cinder-operator-controller-manager-984cd4dcf-75kt6/logs/manager.log.gz
2026-03-14 10:05:41.330128 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/designate-operator-controller-manager-66d56f6ff4-mxn5j/
2026-03-14 10:05:41.330145 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/designate-operator-controller-manager-66d56f6ff4-mxn5j/designate-operator-controller-manager-66d56f6ff4-mxn5j-describe
2026-03-14 10:05:41.330162 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/designate-operator-controller-manager-66d56f6ff4-mxn5j/logs/
2026-03-14 10:05:41.330179 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/designate-operator-controller-manager-66d56f6ff4-mxn5j/logs/manager.log
2026-03-14 10:05:41.330196 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/glance-operator-controller-manager-5964f64c48-g29z7/
2026-03-14 10:05:41.330214 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/glance-operator-controller-manager-5964f64c48-g29z7/glance-operator-controller-manager-5964f64c48-g29z7-describe
2026-03-14 10:05:41.330232 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/glance-operator-controller-manager-5964f64c48-g29z7/logs/
2026-03-14 10:05:41.330249 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/glance-operator-controller-manager-5964f64c48-g29z7/logs/manager.log
2026-03-14 10:05:41.330270 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/heat-operator-controller-manager-77b6666d85-czghc/
2026-03-14 10:05:41.330289 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/heat-operator-controller-manager-77b6666d85-czghc/heat-operator-controller-manager-77b6666d85-czghc-describe
2026-03-14 10:05:41.330306 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/heat-operator-controller-manager-77b6666d85-czghc/logs/
2026-03-14 10:05:41.330324 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/heat-operator-controller-manager-77b6666d85-czghc/logs/manager.log
2026-03-14 10:05:41.330341 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/horizon-operator-controller-manager-6d9d6b584d-bmb27/
2026-03-14 10:05:41.330358 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/horizon-operator-controller-manager-6d9d6b584d-bmb27/horizon-operator-controller-manager-6d9d6b584d-bmb27-describe
2026-03-14 10:05:41.330376 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/horizon-operator-controller-manager-6d9d6b584d-bmb27/logs/
2026-03-14 10:05:41.330393 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/horizon-operator-controller-manager-6d9d6b584d-bmb27/logs/manager.log
2026-03-14 10:05:41.330410 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/infra-operator-controller-manager-54dc5b8f8d-ffxmw/
2026-03-14 10:05:41.330427 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/infra-operator-controller-manager-54dc5b8f8d-ffxmw/infra-operator-controller-manager-54dc5b8f8d-ffxmw-describe
2026-03-14 10:05:41.330444 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/infra-operator-controller-manager-54dc5b8f8d-ffxmw/logs/
2026-03-14 10:05:41.330461 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/infra-operator-controller-manager-54dc5b8f8d-ffxmw/logs/manager.log
2026-03-14 10:05:41.330482 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ironic-operator-controller-manager-5bc894d9b-8wvhp/
2026-03-14 10:05:41.330499 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ironic-operator-controller-manager-5bc894d9b-8wvhp/ironic-operator-controller-manager-5bc894d9b-8wvhp-describe
2026-03-14 10:05:41.330516 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ironic-operator-controller-manager-5bc894d9b-8wvhp/logs/
2026-03-14 10:05:41.330534 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ironic-operator-controller-manager-5bc894d9b-8wvhp/logs/manager.log
2026-03-14 10:05:41.330551 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/keystone-operator-controller-manager-684f77d66d-7v8h2/
2026-03-14 10:05:41.330568 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/keystone-operator-controller-manager-684f77d66d-7v8h2/keystone-operator-controller-manager-684f77d66d-7v8h2-describe
2026-03-14 10:05:41.330586 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/keystone-operator-controller-manager-684f77d66d-7v8h2/logs/
2026-03-14 10:05:41.330604 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/keystone-operator-controller-manager-684f77d66d-7v8h2/logs/manager.log
2026-03-14 10:05:41.330621 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/manila-operator-controller-manager-57b484b4df-fqfbh/
2026-03-14 10:05:41.330639 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/manila-operator-controller-manager-57b484b4df-fqfbh/manila-operator-controller-manager-57b484b4df-fqfbh-describe
2026-03-14 10:05:41.330656 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/manila-operator-controller-manager-57b484b4df-fqfbh/logs/
2026-03-14 10:05:41.330674 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/manila-operator-controller-manager-57b484b4df-fqfbh/logs/manager.log
2026-03-14 10:05:41.330717 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/mariadb-operator-controller-manager-5b6b6b4c9f-pdpx8/
2026-03-14 10:05:41.330737 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/mariadb-operator-controller-manager-5b6b6b4c9f-pdpx8/mariadb-operator-controller-manager-5b6b6b4c9f-pdpx8-describe
2026-03-14 10:05:41.330754 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/mariadb-operator-controller-manager-5b6b6b4c9f-pdpx8/logs/
2026-03-14 10:05:41.330772 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/mariadb-operator-controller-manager-5b6b6b4c9f-pdpx8/logs/manager.log
2026-03-14 10:05:41.330790 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/neutron-operator-controller-manager-776c5696bf-9dlkn/
2026-03-14 10:05:41.330807 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/neutron-operator-controller-manager-776c5696bf-9dlkn/neutron-operator-controller-manager-776c5696bf-9dlkn-describe
2026-03-14 10:05:41.330824 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/neutron-operator-controller-manager-776c5696bf-9dlkn/logs/
2026-03-14 10:05:41.330842 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/neutron-operator-controller-manager-776c5696bf-9dlkn/logs/manager.log
2026-03-14 10:05:41.330860 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/nova-operator-controller-manager-7f84474648-g59td/
2026-03-14 10:05:41.330877 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/nova-operator-controller-manager-7f84474648-g59td/nova-operator-controller-manager-7f84474648-g59td-describe
2026-03-14 10:05:41.330895 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/nova-operator-controller-manager-7f84474648-g59td/logs/
2026-03-14 10:05:41.330912 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/nova-operator-controller-manager-7f84474648-g59td/logs/manager.log
2026-03-14 10:05:41.330930 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/octavia-operator-controller-manager-5f4f55cb5c-zrfcm/
2026-03-14 10:05:41.330951 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/octavia-operator-controller-manager-5f4f55cb5c-zrfcm/octavia-operator-controller-manager-5f4f55cb5c-zrfcm-describe
2026-03-14 10:05:41.330969 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/octavia-operator-controller-manager-5f4f55cb5c-zrfcm/logs/
2026-03-14 10:05:41.330987 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/octavia-operator-controller-manager-5f4f55cb5c-zrfcm/logs/manager.log
2026-03-14 10:05:41.331004 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-baremetal-operator-controller-manager-557ccf57b7slhjt/
2026-03-14 10:05:41.331021 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-baremetal-operator-controller-manager-557ccf57b7slhjt/openstack-baremetal-operator-controller-manager-557ccf57b7slhjt-describe
2026-03-14 10:05:41.331038 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-baremetal-operator-controller-manager-557ccf57b7slhjt/logs/
2026-03-14 10:05:41.331056 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-baremetal-operator-controller-manager-557ccf57b7slhjt/logs/manager.log
2026-03-14 10:05:41.331090 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-controller-manager-59cdb7596d-l4ggb/
2026-03-14 10:05:41.331108 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-controller-manager-59cdb7596d-l4ggb/openstack-operator-controller-manager-59cdb7596d-l4ggb-describe
2026-03-14 10:05:41.331128 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-controller-manager-59cdb7596d-l4ggb/logs/
2026-03-14 10:05:41.331146 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-controller-manager-59cdb7596d-l4ggb/logs/manager.log.gz
2026-03-14 10:05:41.331164 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-index-6s6vv/
2026-03-14 10:05:41.331185 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-index-6s6vv/openstack-operator-index-6s6vv-describe
2026-03-14 10:05:41.331203 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-index-6s6vv/logs/
2026-03-14 10:05:41.331220 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-index-6s6vv/logs/registry-server.log
2026-03-14 10:05:41.331237 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ovn-operator-controller-manager-bbc5b68f9-xctxr/
2026-03-14 10:05:41.331255 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ovn-operator-controller-manager-bbc5b68f9-xctxr/ovn-operator-controller-manager-bbc5b68f9-xctxr-describe
2026-03-14 10:05:41.331273 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ovn-operator-controller-manager-bbc5b68f9-xctxr/logs/
2026-03-14 10:05:41.331291 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ovn-operator-controller-manager-bbc5b68f9-xctxr/logs/manager.log
2026-03-14 10:05:41.331308 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/placement-operator-controller-manager-574d45c66c-tx8dt/
2026-03-14 10:05:41.331326 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/placement-operator-controller-manager-574d45c66c-tx8dt/placement-operator-controller-manager-574d45c66c-tx8dt-describe
2026-03-14 10:05:41.331343 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/placement-operator-controller-manager-574d45c66c-tx8dt/logs/
2026-03-14 10:05:41.331360 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/placement-operator-controller-manager-574d45c66c-tx8dt/logs/manager.log
2026-03-14 10:05:41.331378 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/rabbitmq-cluster-operator-manager-668c99d594-qjs78/
2026-03-14 10:05:41.331399 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/rabbitmq-cluster-operator-manager-668c99d594-qjs78/rabbitmq-cluster-operator-manager-668c99d594-qjs78-describe
2026-03-14 10:05:41.331417 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/rabbitmq-cluster-operator-manager-668c99d594-qjs78/logs/
2026-03-14 10:05:41.331434 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/rabbitmq-cluster-operator-manager-668c99d594-qjs78/logs/operator.log
2026-03-14 10:05:41.331451 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/swift-operator-controller-manager-7f9cc5dd44-kbcdm/
2026-03-14 10:05:41.331469 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/swift-operator-controller-manager-7f9cc5dd44-kbcdm/swift-operator-controller-manager-7f9cc5dd44-kbcdm-describe
2026-03-14 10:05:41.331486 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/swift-operator-controller-manager-7f9cc5dd44-kbcdm/logs/
2026-03-14 10:05:41.331504 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/swift-operator-controller-manager-7f9cc5dd44-kbcdm/logs/manager.log
2026-03-14 10:05:41.331523 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/telemetry-operator-controller-manager-6854b8b9d9-vgstg/
2026-03-14 10:05:41.331540 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/telemetry-operator-controller-manager-6854b8b9d9-vgstg/telemetry-operator-controller-manager-6854b8b9d9-vgstg-describe
2026-03-14 10:05:41.331559 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/telemetry-operator-controller-manager-6854b8b9d9-vgstg/logs/
2026-03-14 10:05:41.331577 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/telemetry-operator-controller-manager-6854b8b9d9-vgstg/logs/manager.log.gz
2026-03-14 10:05:41.331595 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/test-operator-controller-manager-5c5cb9c4d7-9nkkk/
2026-03-14 10:05:41.331613 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/test-operator-controller-manager-5c5cb9c4d7-9nkkk/test-operator-controller-manager-5c5cb9c4d7-9nkkk-describe
2026-03-14 10:05:41.331638 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/test-operator-controller-manager-5c5cb9c4d7-9nkkk/logs/
2026-03-14 10:05:41.331657 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/test-operator-controller-manager-5c5cb9c4d7-9nkkk/logs/manager.log
2026-03-14 10:05:41.331770 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-controller-manager-66c89757b4-kgcsg/
2026-03-14 10:05:41.331813 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-controller-manager-66c89757b4-kgcsg/watcher-operator-controller-manager-66c89757b4-kgcsg-describe
2026-03-14 10:05:41.331835 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-controller-manager-66c89757b4-kgcsg/logs/
2026-03-14 10:05:41.331853 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-controller-manager-66c89757b4-kgcsg/logs/manager.log.gz
2026-03-14 10:05:41.331871 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-index-7zk59/
2026-03-14 10:05:41.331890 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-index-7zk59/watcher-operator-index-7zk59-describe
2026-03-14 10:05:41.331907 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-index-7zk59/logs/
2026-03-14 10:05:41.331924 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-index-7zk59/logs/registry-server.log
2026-03-14 10:05:41.331942 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/
2026-03-14 10:05:41.331960 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/barbican-operator-controller-manager-d47688694.yaml
2026-03-14 10:05:41.331982 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/cinder-operator-controller-manager-984cd4dcf.yaml
2026-03-14 10:05:41.332004 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/designate-operator-controller-manager-66d56f6ff4.yaml
2026-03-14 10:05:41.332028 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/glance-operator-controller-manager-5964f64c48.yaml
2026-03-14 10:05:41.332052 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/heat-operator-controller-manager-77b6666d85.yaml
2026-03-14 10:05:41.332076 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/horizon-operator-controller-manager-6d9d6b584d.yaml
2026-03-14 10:05:41.332103 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/infra-operator-controller-manager-54dc5b8f8d.yaml
2026-03-14 10:05:41.332127 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/ironic-operator-controller-manager-5bc894d9b.yaml
2026-03-14 10:05:41.332146 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/keystone-operator-controller-manager-684f77d66d.yaml
2026-03-14 10:05:41.332164 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/manila-operator-controller-manager-57b484b4df.yaml
2026-03-14 10:05:41.332182 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/mariadb-operator-controller-manager-5b6b6b4c9f.yaml
2026-03-14 10:05:41.332202 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/neutron-operator-controller-manager-776c5696bf.yaml
2026-03-14 10:05:41.332222 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/nova-operator-controller-manager-7f84474648.yaml
2026-03-14 10:05:41.332238 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/octavia-operator-controller-manager-5f4f55cb5c.yaml
2026-03-14 10:05:41.332260 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/openstack-baremetal-operator-controller-manager-557ccf57b7.yaml
2026-03-14 10:05:41.332278 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/openstack-operator-controller-init-6bdb46c895.yaml
2026-03-14 10:05:41.332294 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/openstack-operator-controller-manager-59cdb7596d.yaml
2026-03-14 10:05:41.332312 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/ovn-operator-controller-manager-bbc5b68f9.yaml
2026-03-14 10:05:41.332329 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/placement-operator-controller-manager-574d45c66c.yaml
2026-03-14 10:05:41.332346 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/rabbitmq-cluster-operator-manager-668c99d594.yaml
2026-03-14 10:05:41.332364 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/swift-operator-controller-manager-7f9cc5dd44.yaml
2026-03-14 10:05:41.332381 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/telemetry-operator-controller-manager-6854b8b9d9.yaml
2026-03-14 10:05:41.332399 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/test-operator-controller-manager-5c5cb9c4d7.yaml
2026-03-14 10:05:41.332417 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/watcher-operator-controller-manager-66c89757b4.yaml
2026-03-14 10:05:41.332434 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/watcher-operator-controller-manager-6d87bdbc68.yaml
2026-03-14 10:05:41.332452 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/routes/
2026-03-14 10:05:41.332470 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/
2026-03-14 10:05:41.332491 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/barbican/
2026-03-14 10:05:41.332509 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/barbican/barbican-operator-metrics-server-cert.yaml
2026-03-14 10:05:41.332527 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/cinder/
2026-03-14 10:05:41.332544 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/cinder/cinder-operator-metrics-server-cert.yaml
2026-03-14 10:05:41.332562 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/designate/
2026-03-14 10:05:41.332580 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/designate/designate-operator-metrics-server-cert.yaml
2026-03-14 10:05:41.332597 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/glance/
2026-03-14 10:05:41.332614 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/glance/glance-operator-metrics-server-cert.yaml
2026-03-14 10:05:41.332632 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/heat/
2026-03-14 10:05:41.332649 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/heat/heat-operator-metrics-server-cert.yaml
2026-03-14 10:05:41.332666 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/horizon/
2026-03-14 10:05:41.332707 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/horizon/horizon-operator-metrics-server-cert.yaml
2026-03-14 10:05:41.332726 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/ironic/
2026-03-14 10:05:41.332743 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/ironic/ironic-operator-metrics-server-cert.yaml
2026-03-14 10:05:41.332765 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/keystone/
2026-03-14 10:05:41.332783 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/keystone/keystone-operator-metrics-server-cert.yaml
2026-03-14 10:05:41.332800 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/manila/
2026-03-14 10:05:41.332818 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/manila/manila-operator-metrics-server-cert.yaml
2026-03-14 10:05:41.332835 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/mariadb/
2026-03-14 10:05:41.332852 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/mariadb/mariadb-operator-metrics-server-cert.yaml
2026-03-14 10:05:41.332869 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/neutron/
2026-03-14 10:05:41.332887 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/neutron/neutron-operator-metrics-server-cert.yaml
2026-03-14 10:05:41.332905 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/nova/
2026-03-14 10:05:41.332923 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/nova/nova-operator-metrics-server-cert.yaml
2026-03-14 10:05:41.332940 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/octavia/
2026-03-14 10:05:41.332958 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/octavia/octavia-operator-metrics-server-cert.yaml
2026-03-14 10:05:41.332976 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/ovn/
2026-03-14 10:05:41.332994 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/ovn/ovn-operator-metrics-server-cert.yaml
2026-03-14 10:05:41.333015 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/placement/
2026-03-14 10:05:41.333033 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/placement/placement-operator-metrics-server-cert.yaml
2026-03-14 10:05:41.333051 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/swift/
2026-03-14 10:05:41.333069 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/swift/swift-operator-metrics-server-cert.yaml
2026-03-14 10:05:41.333086 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/telemetry/
2026-03-14 10:05:41.333103 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/telemetry/telemetry-operator-metrics-server-cert.yaml
2026-03-14 10:05:41.333121 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/watcher/
2026-03-14 10:05:41.333138 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/watcher/watcher-operator-controller-manager-service-cert.yaml
2026-03-14 10:05:41.333156 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/watcher/watcher-operator-metrics-server-cert.yaml
2026-03-14 10:05:41.333173 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/
2026-03-14 10:05:41.333191 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/barbican-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333207 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/cinder-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333225 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/designate-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333242 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/glance-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333264 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/heat-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333281 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/horizon-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333298 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/infra-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333316 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/infra-operator-webhook-service.yaml
2026-03-14 10:05:41.333334 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/ironic-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333351 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/keystone-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333368 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/manila-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333385 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/mariadb-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333403 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/neutron-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333420 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/nova-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333439 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/octavia-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333456 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/openstack-baremetal-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333473 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/openstack-baremetal-operator-webhook-service.yaml
2026-03-14 10:05:41.333494 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/openstack-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333513 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/openstack-operator-index.yaml
2026-03-14 10:05:41.333530 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/openstack-operator-webhook-service.yaml
2026-03-14 10:05:41.333547 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/ovn-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333565 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/placement-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333583 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/rabbitmq-cluster-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333601 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/swift-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333618 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/telemetry-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333634 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/test-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333651 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/watcher-operator-controller-manager-metrics-service.yaml
2026-03-14 10:05:41.333669 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/watcher-operator-controller-manager-service.yaml
2026-03-14 10:05:41.333705 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/watcher-operator-index.yaml
2026-03-14 10:05:41.333724 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/watcher-operator-webhook-service.yaml
2026-03-14 10:05:41.333746 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/statefulset/
2026-03-14 10:05:41.333764 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/subscriptions/
2026-03-14 10:05:41.333781 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/subscriptions/openstack-operator
2026-03-14 10:05:41.333799 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/subscriptions/watcher-operator
2026-03-14 10:05:41.333817 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/
2026-03-14 10:05:41.333835 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/all_resources.log
2026-03-14 10:05:41.333855 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/events.log
2026-03-14 10:05:41.333873 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/nad.log
2026-03-14 10:05:41.333891 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/pvc.log
2026-03-14 10:05:41.333908 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/buildconfig/
2026-03-14 10:05:41.333926 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/cert/
2026-03-14 10:05:41.333943 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/certificaterequests/
2026-03-14 10:05:41.333960 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/configmaps/
2026-03-14 10:05:41.333977 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/cronjobs/
2026-03-14 10:05:41.333993 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/
2026-03-14 10:05:41.334014 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/
2026-03-14 10:05:41.334032 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/ctlplane.yaml
2026-03-14 10:05:41.334049 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/datacentre.yaml
2026-03-14 10:05:41.334066 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/designate.yaml
2026-03-14 10:05:41.334083 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/designateext.yaml
2026-03-14 10:05:41.334100 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/internalapi.yaml
2026-03-14 10:05:41.334117 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/octavia.yaml
2026-03-14 10:05:41.334135 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/storage.yaml
2026-03-14 10:05:41.334152 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/storagemgmt.yaml
2026-03-14 10:05:41.334169 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/tenant.yaml
2026-03-14 10:05:41.334186 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/daemonset/
2026-03-14 10:05:41.334203 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/deployments/
2026-03-14 10:05:41.334220 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/imagestream/
2026-03-14 10:05:41.334237 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/installplans/
2026-03-14 10:05:41.334258 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/issuer/
2026-03-14 10:05:41.334276 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/jobs/
2026-03-14 10:05:41.334292 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/poddisruptionbudgets/
2026-03-14 10:05:41.334310 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/pods/
2026-03-14 10:05:41.334327 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/pods/-describe
2026-03-14 10:05:41.334345 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/pods/logs/
2026-03-14 10:05:41.334362 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/replicaset/
2026-03-14 10:05:41.334380 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/routes/
2026-03-14 10:05:41.334397 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/services/
2026-03-14 10:05:41.334414 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/statefulset/
2026-03-14 10:05:41.334431 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/subscriptions/
2026-03-14 10:05:41.334449 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/
2026-03-14 10:05:41.334467 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/all_resources.log
2026-03-14 10:05:41.334484 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/events.log
2026-03-14 10:05:41.334501 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/nad.log
2026-03-14 10:05:41.334517 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pvc.log
2026-03-14 10:05:41.334539 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/buildconfig/
2026-03-14 10:05:41.334557 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/
2026-03-14 10:05:41.334573 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/ceilometer-internal-svc.yaml
2026-03-14 10:05:41.334589 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/cinder-internal-svc.yaml
2026-03-14 10:05:41.334607 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/galera-openstack-svc.yaml
2026-03-14 10:05:41.334624 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/keystone-internal-svc.yaml
2026-03-14 10:05:41.334641 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/keystone-public-route.yaml
2026-03-14 10:05:41.334657 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/keystone-public-svc.yaml
2026-03-14 10:05:41.334673 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/kube-state-metrics-svc.yaml
2026-03-14 10:05:41.334710 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/memcached-mtls.yaml
2026-03-14 10:05:41.334728 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/memcached-svc.yaml
2026-03-14 10:05:41.334745 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/metric-storage-prometheus-svc.yaml
2026-03-14 10:05:41.334763 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/rabbitmq-notifications-svc.yaml
2026-03-14 10:05:41.334779 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/rabbitmq-svc.yaml
2026-03-14 10:05:41.334800 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/rootca-internal.yaml
2026-03-14 10:05:41.334818 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/rootca-libvirt.yaml
2026-03-14 10:05:41.334834 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/rootca-ovn.yaml
2026-03-14 10:05:41.334851 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/rootca-public.yaml
2026-03-14 10:05:41.334866 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/
2026-03-14 10:05:41.334883 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/ceilometer-internal-svc-1.yaml
2026-03-14 10:05:41.334900 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/cinder-internal-svc-1.yaml
2026-03-14 10:05:41.334917 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/galera-openstack-svc-1.yaml
2026-03-14 10:05:41.334934 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/keystone-internal-svc-1.yaml
2026-03-14 10:05:41.334950 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/keystone-public-route-1.yaml
2026-03-14 10:05:41.334968 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/keystone-public-svc-1.yaml
2026-03-14 10:05:41.334985 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/kube-state-metrics-svc-1.yaml
2026-03-14 10:05:41.335002 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/memcached-mtls-1.yaml
2026-03-14 10:05:41.335019 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/memcached-svc-1.yaml
2026-03-14 10:05:41.335042 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/metric-storage-prometheus-svc-1.yaml
2026-03-14 10:05:41.335073 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/rabbitmq-notifications-svc-1.yaml
2026-03-14 10:05:41.335093 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/rabbitmq-svc-1.yaml
2026-03-14 10:05:41.335110 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/rootca-internal-1.yaml
2026-03-14 10:05:41.335131 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/rootca-libvirt-1.yaml
2026-03-14 10:05:41.335148 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/rootca-ovn-1.yaml
2026-03-14 10:05:41.335166 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/rootca-public-1.yaml
2026-03-14 10:05:41.335184 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/
2026-03-14 10:05:41.335203 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/appcred-watcher-pre.yaml
2026-03-14 10:05:41.335222 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/infra-instanceha-config.yaml
2026-03-14 10:05:41.335239 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/prometheus-metric-storage-rulefiles-0.yaml
2026-03-14 10:05:41.335257 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/rabbitmq-config-data.yaml
2026-03-14 10:05:41.335274 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/rabbitmq-notifications-config-data.yaml
2026-03-14 10:05:41.335292 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/rabbitmq-notifications-plugins-conf.yaml
2026-03-14 10:05:41.335314 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/rabbitmq-notifications-server-conf.yaml
2026-03-14 10:05:41.335331 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/rabbitmq-plugins-conf.yaml
2026-03-14 10:05:41.335348 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/rabbitmq-server-conf.yaml
2026-03-14 10:05:41.335366 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cronjobs/
2026-03-14 10:05:41.335382 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cronjobs/keystone-cron.yaml
2026-03-14 10:05:41.335400 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/
2026-03-14 10:05:41.335416 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/alertmanagers.monitoring.rhobs/
2026-03-14 10:05:41.335433 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/alertmanagers.monitoring.rhobs/metric-storage.yaml
2026-03-14 10:05:41.335450 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/ceilometers.telemetry.openstack.org/
2026-03-14 10:05:41.335468 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/ceilometers.telemetry.openstack.org/ceilometer.yaml
2026-03-14 10:05:41.335484 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/galeras.mariadb.openstack.org/
2026-03-14 10:05:41.335501 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/galeras.mariadb.openstack.org/openstack.yaml
2026-03-14 10:05:41.335519 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/keystoneapis.keystone.openstack.org/
2026-03-14 10:05:41.335542 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/keystoneapis.keystone.openstack.org/keystone.yaml
2026-03-14 10:05:41.335560 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/keystoneservices.keystone.openstack.org/
2026-03-14 10:05:41.335577 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/keystoneservices.keystone.openstack.org/ceilometer.yaml
2026-03-14 10:05:41.335595 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/mariadbaccounts.mariadb.openstack.org/
2026-03-14 10:05:41.335612 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/mariadbaccounts.mariadb.openstack.org/keystone.yaml
2026-03-14 10:05:41.335629 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/mariadbaccounts.mariadb.openstack.org/openstack-mariadb-root.yaml
2026-03-14 10:05:41.335647 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/mariadbdatabases.mariadb.openstack.org/
2026-03-14 10:05:41.335665 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/mariadbdatabases.mariadb.openstack.org/keystone.yaml
2026-03-14 10:05:41.335700 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/memcacheds.memcached.openstack.org/
2026-03-14 10:05:41.335720 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/memcacheds.memcached.openstack.org/memcached.yaml
2026-03-14 10:05:41.335738 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/metricstorages.telemetry.openstack.org/
2026-03-14 10:05:41.335755 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/metricstorages.telemetry.openstack.org/metric-storage.yaml
2026-03-14 10:05:41.335772 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/monitoringstacks.monitoring.rhobs/
2026-03-14 10:05:41.335790 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/monitoringstacks.monitoring.rhobs/metric-storage.yaml
2026-03-14 10:05:41.335811 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/openstackclients.client.openstack.org/
2026-03-14 10:05:41.335828 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/openstackclients.client.openstack.org/openstackclient.yaml
2026-03-14 10:05:41.335846 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/openstackcontrolplanes.core.openstack.org/
2026-03-14 10:05:41.335864 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/openstackcontrolplanes.core.openstack.org/openstack.yaml
2026-03-14 10:05:41.335881 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/openstackversions.core.openstack.org/
2026-03-14 10:05:41.335898 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/openstackversions.core.openstack.org/openstack.yaml
2026-03-14 10:05:41.335916 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/prometheuses.monitoring.rhobs/
2026-03-14 10:05:41.335934 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/prometheuses.monitoring.rhobs/metric-storage.yaml
2026-03-14 10:05:41.335951 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/prometheusrules.monitoring.rhobs/
2026-03-14 10:05:41.335969 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/prometheusrules.monitoring.rhobs/metric-storage.yaml
2026-03-14 10:05:41.335986 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/rabbitmqclusters.rabbitmq.com/
2026-03-14 10:05:41.336004 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/rabbitmqclusters.rabbitmq.com/rabbitmq-notifications.yaml
2026-03-14 10:05:41.336022 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/rabbitmqclusters.rabbitmq.com/rabbitmq.yaml
2026-03-14 10:05:41.336042 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/rabbitmqs.rabbitmq.openstack.org/
2026-03-14 10:05:41.336060 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/rabbitmqs.rabbitmq.openstack.org/rabbitmq-notifications.yaml
2026-03-14 10:05:41.336077 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/rabbitmqs.rabbitmq.openstack.org/rabbitmq.yaml
2026-03-14 10:05:41.336094 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/
2026-03-14 10:05:41.336110 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ceilometer-compute-prom-exporter-tls.yaml
2026-03-14 10:05:41.336128 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ceilometer-compute-prom-exporter.yaml
2026-03-14 10:05:41.336145 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ceilometer-ipmi-prom-exporter-tls.yaml
2026-03-14 10:05:41.336161 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ceilometer-ipmi-prom-exporter.yaml
2026-03-14 10:05:41.336178 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ceilometer.yaml
2026-03-14 10:05:41.336195 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-kepler.yaml
2026-03-14 10:05:41.336212 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ksm.yaml
2026-03-14 10:05:41.336229 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-node-exporter-tls.yaml
2026-03-14 10:05:41.336246 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-node-exporter.yaml
2026-03-14 10:05:41.336266 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-openstack-network-exporter-tls.yaml
2026-03-14 10:05:41.336284 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-openstack-network-exporter.yaml
2026-03-14 10:05:41.336302 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-podman-exporter-tls.yaml
2026-03-14 10:05:41.336319 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-podman-exporter.yaml
2026-03-14 10:05:41.336337 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-rabbitmq-notifications.yaml
2026-03-14 10:05:41.336354 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-rabbitmq.yaml
2026-03-14 10:05:41.336372 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/telemetries.telemetry.openstack.org/
2026-03-14 10:05:41.336389 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/telemetries.telemetry.openstack.org/telemetry.yaml
2026-03-14 10:05:41.336408 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/transporturls.rabbitmq.openstack.org/
2026-03-14 10:05:41.336425 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/transporturls.rabbitmq.openstack.org/ceilometer-notifications-transport.yaml
2026-03-14 10:05:41.336442 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/transporturls.rabbitmq.openstack.org/keystone-keystone-transport.yaml
2026-03-14 10:05:41.336459 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/daemonset/
2026-03-14 10:05:41.336476 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/deployments/
2026-03-14 10:05:41.336497 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/deployments/keystone.yaml
2026-03-14 10:05:41.336515 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/imagestream/
2026-03-14 10:05:41.336532 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/installplans/
2026-03-14 10:05:41.336548 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/issuer/
2026-03-14 10:05:41.336565 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/issuer/rootca-internal.yaml
2026-03-14 10:05:41.336582 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/issuer/rootca-libvirt.yaml
2026-03-14 10:05:41.336600 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/issuer/rootca-ovn.yaml
2026-03-14 10:05:41.336617 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/issuer/rootca-public.yaml
2026-03-14 10:05:41.336635 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/issuer/selfsigned-issuer.yaml
2026-03-14 10:05:41.336652 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/jobs/
2026-03-14 10:05:41.336670 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/poddisruptionbudgets/
2026-03-14 10:05:41.336707 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/poddisruptionbudgets/metric-storage-alertmanager.yaml
2026-03-14 10:05:41.336726 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/
2026-03-14 10:05:41.336744 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/
2026-03-14 10:05:41.336766 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/alertmanager-metric-storage-0-describe
2026-03-14 10:05:41.336784 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/logs/
2026-03-14 10:05:41.336802 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/logs/alertmanager.log
2026-03-14 10:05:41.336819 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/logs/config-reloader.log
2026-03-14 10:05:41.336837 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/logs/init-config-reloader.log
2026-03-14 10:05:41.336855 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/
2026-03-14 10:05:41.336872 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/ceilometer-0-describe
2026-03-14 10:05:41.336890 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/logs/
2026-03-14 10:05:41.336907 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/logs/ceilometer-central-agent.log
2026-03-14 10:05:41.336924 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/logs/ceilometer-notification-agent.log
2026-03-14 10:05:41.336942 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/logs/proxy-httpd.log
2026-03-14 10:05:41.336960 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/logs/sg-core.log
2026-03-14 10:05:41.336977 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/keystone-7f5697c5fb-hqksz/
2026-03-14 10:05:41.336994 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/keystone-7f5697c5fb-hqksz/keystone-7f5697c5fb-hqksz-describe
2026-03-14 10:05:41.337016 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/keystone-7f5697c5fb-hqksz/logs/
2026-03-14 10:05:41.337033 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/keystone-7f5697c5fb-hqksz/logs/keystone-api.log
2026-03-14 10:05:41.337051 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/kube-state-metrics-0/
2026-03-14 10:05:41.337068 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/kube-state-metrics-0/kube-state-metrics-0-describe
2026-03-14 10:05:41.337086 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/kube-state-metrics-0/logs/
2026-03-14 10:05:41.337103 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/kube-state-metrics-0/logs/kube-state-metrics.log
2026-03-14 10:05:41.337120 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/memcached-0/
2026-03-14 10:05:41.337137 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/memcached-0/memcached-0-describe
2026-03-14 10:05:41.337154 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/memcached-0/logs/
2026-03-14 10:05:41.337171 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/memcached-0/logs/memcached.log.gz
2026-03-14 10:05:41.337188 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstack-galera-0/
2026-03-14 10:05:41.337205 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstack-galera-0/openstack-galera-0-describe
2026-03-14 10:05:41.337222 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstack-galera-0/logs/
2026-03-14 10:05:41.337243 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstack-galera-0/logs/galera.log
2026-03-14 10:05:41.337261 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstack-galera-0/logs/mysql-bootstrap.log
2026-03-14 10:05:41.337278 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstackclient/
2026-03-14 10:05:41.337295 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstackclient/openstackclient-describe
2026-03-14 10:05:41.337312 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstackclient/logs/
2026-03-14 10:05:41.337330 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstackclient/logs/openstackclient.log
2026-03-14 10:05:41.337347 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/
2026-03-14 10:05:41.337364 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/prometheus-metric-storage-0-describe
2026-03-14 10:05:41.337381 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/logs/
2026-03-14 10:05:41.337398 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/logs/config-reloader.log
2026-03-14 10:05:41.337415 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/logs/init-config-reloader.log
2026-03-14 10:05:41.337432 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/logs/prometheus.log
2026-03-14 10:05:41.337449 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/logs/thanos-sidecar.log
2026-03-14 10:05:41.337465 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-notifications-server-0/
2026-03-14 10:05:41.337486 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-notifications-server-0/rabbitmq-notifications-server-0-describe
2026-03-14 10:05:41.337503 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-notifications-server-0/logs/
2026-03-14 10:05:41.337522 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-notifications-server-0/logs/rabbitmq.log
2026-03-14 10:05:41.337540 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-notifications-server-0/logs/setup-container.log
2026-03-14 10:05:41.337558 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-server-0/
2026-03-14 10:05:41.337576 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-server-0/rabbitmq-server-0-describe
2026-03-14 10:05:41.337593 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-server-0/logs/
2026-03-14 10:05:41.337611 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-server-0/logs/rabbitmq.log
2026-03-14 10:05:41.337628 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-server-0/logs/setup-container.log
2026-03-14 10:05:41.337645 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/replicaset/
2026-03-14 10:05:41.337662 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/replicaset/keystone-577555cd46.yaml
2026-03-14 10:05:41.337694 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/replicaset/keystone-7f5697c5fb.yaml
2026-03-14 10:05:41.337716 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/routes/
2026-03-14 10:05:41.337734 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/routes/keystone-public.yaml
2026-03-14 10:05:41.337755 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/
2026-03-14 10:05:41.337774 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/
2026-03-14 10:05:41.337791 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-compute-config-data.yaml
2026-03-14 10:05:41.337808 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-compute-config-data.yaml-ceilometer.conf
2026-03-14 10:05:41.337826 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-compute-config-data.yaml-custom.conf
2026-03-14 10:05:41.337843 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-compute-scripts.yaml
2026-03-14 10:05:41.337862 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-config-data.yaml
2026-03-14 10:05:41.337880 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-config-data.yaml-ceilometer.conf
2026-03-14 10:05:41.337898 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-config-data.yaml-custom.conf
2026-03-14 10:05:41.337916 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-config-data.yaml-httpd.conf
2026-03-14 10:05:41.337934 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-config-data.yaml-ssl.conf
2026-03-14 10:05:41.337952 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-ipmi-config-data.yaml
2026-03-14 10:05:41.337969 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-ipmi-config-data.yaml-ceilometer.conf
2026-03-14 10:05:41.337990 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-ipmi-config-data.yaml-custom.conf
2026-03-14 10:05:41.338008 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-ipmi-scripts.yaml
2026-03-14 10:05:41.338025 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-scripts.yaml
2026-03-14 10:05:41.338041 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/cert-ceilometer-internal-svc.yaml
2026-03-14 10:05:41.338058 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/rabbitmq-transport-url-ceilometer-notifications-transport.yaml
2026-03-14 10:05:41.338075 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/cinder/
2026-03-14 10:05:41.338092 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/cinder/cert-cinder-internal-svc.yaml
2026-03-14 10:05:41.338110 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/cinder/cert-cinder-public-route.yaml
2026-03-14 10:05:41.338127 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/cinder/cert-cinder-public-svc.yaml
2026-03-14 10:05:41.338144 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/
2026-03-14 10:05:41.338161 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/cert-keystone-internal-svc.yaml
2026-03-14 10:05:41.338177 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/cert-keystone-public-route.yaml
2026-03-14 10:05:41.338194 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/cert-keystone-public-svc.yaml
2026-03-14 10:05:41.338215 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone-config-data.yaml
2026-03-14 10:05:41.338233 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone-config-data.yaml-httpd.conf
2026-03-14 10:05:41.338250 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone-config-data.yaml-keystone.conf
2026-03-14 10:05:41.338267 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone-config-data.yaml-ssl.conf
2026-03-14 10:05:41.338284 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone-db-secret.yaml
2026-03-14 10:05:41.338301 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone-scripts.yaml
2026-03-14 10:05:41.338319 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone.yaml
2026-03-14 10:05:41.338335 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/rabbitmq-transport-url-keystone-keystone-transport.yaml
2026-03-14 10:05:41.338353 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/mariadb/
2026-03-14 10:05:41.338371 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/mariadb/openstack-mariadb-root-db-secret.yaml
2026-03-14 10:05:41.338388 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/
2026-03-14 10:05:41.338405 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/alertmanager-metric-storage-cluster-tls-config.yaml
2026-03-14 10:05:41.338422 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/alertmanager-metric-storage-generated.yaml
2026-03-14 10:05:41.338439 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/alertmanager-metric-storage-tls-assets-0.yaml
2026-03-14 10:05:41.338461 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/alertmanager-metric-storage-web-config.yaml
2026-03-14 10:05:41.338479 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/cert-metric-storage-prometheus-svc.yaml
2026-03-14 10:05:41.338496 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/metric-storage-prometheus-endpoint.yaml
2026-03-14 10:05:41.338514 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/metric-storage-self-scrape.yaml
2026-03-14 10:05:41.338531 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/prometheus-metric-storage-thanos-prometheus-http-client-file.yaml
2026-03-14 10:05:41.338549 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/prometheus-metric-storage-tls-assets-0.yaml
2026-03-14 10:05:41.338566 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/prometheus-metric-storage-web-config.yaml
2026-03-14 10:05:41.338583 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/prometheus-metric-storage.yaml
2026-03-14 10:05:41.338602 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ovn/
2026-03-14 10:05:41.338619 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ovn/rootca-ovn.yaml
2026-03-14 10:05:41.338637 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/
2026-03-14 10:05:41.338654 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/cert-rabbitmq-notifications-svc.yaml
2026-03-14 10:05:41.338671 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/cert-rabbitmq-svc.yaml
2026-03-14 10:05:41.338711 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-default-user.yaml
2026-03-14 10:05:41.338730 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-default-user.yaml-default_user.conf
2026-03-14 10:05:41.338748 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-erlang-cookie.yaml
2026-03-14 10:05:41.338766 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-notifications-default-user.yaml
2026-03-14 10:05:41.338784 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-notifications-default-user.yaml-default_user.conf
2026-03-14 10:05:41.338802 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-notifications-erlang-cookie.yaml
2026-03-14 10:05:41.338819 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-transport-url-ceilometer-notifications-transport.yaml
2026-03-14 10:05:41.338837 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-transport-url-keystone-keystone-transport.yaml
2026-03-14 10:05:41.338855 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/watcher/
2026-03-14 10:05:41.338872 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/watcher/cert-watcher-internal-svc.yaml
2026-03-14 10:05:41.338890 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/watcher/cert-watcher-public-svc.yaml
2026-03-14 10:05:41.338907 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/
2026-03-14 10:05:41.338925 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/alertmanager-operated.yaml
2026-03-14 10:05:41.338945 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/ceilometer-internal.yaml
2026-03-14 10:05:41.338963 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/keystone-internal.yaml
2026-03-14 10:05:41.338980 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/keystone-public.yaml
2026-03-14 10:05:41.338997 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/kube-state-metrics.yaml
2026-03-14 10:05:41.339013 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/memcached.yaml
2026-03-14 10:05:41.339030 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/metric-storage-alertmanager.yaml
2026-03-14 10:05:41.339047 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/metric-storage-prometheus.yaml
2026-03-14 10:05:41.339077 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/metric-storage-thanos-sidecar.yaml
2026-03-14 10:05:41.339099 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/openstack-galera.yaml
2026-03-14 10:05:41.339118 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/openstack.yaml
2026-03-14 10:05:41.339140 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/prometheus-operated.yaml
2026-03-14 10:05:41.339158 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/rabbitmq-nodes.yaml
2026-03-14 10:05:41.339176 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/rabbitmq-notifications-nodes.yaml
2026-03-14 10:05:41.339193 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/rabbitmq-notifications.yaml
2026-03-14 10:05:41.339219 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/rabbitmq.yaml
2026-03-14 10:05:41.339237 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/
2026-03-14 10:05:41.339255 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/alertmanager-metric-storage.yaml
2026-03-14 10:05:41.339273 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/ceilometer.yaml
2026-03-14 10:05:41.339291 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/kube-state-metrics.yaml
2026-03-14 10:05:41.339308 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/memcached.yaml
2026-03-14 10:05:41.339326 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/openstack-galera.yaml
2026-03-14 10:05:41.339342 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/prometheus-metric-storage.yaml
2026-03-14 10:05:41.339358 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/rabbitmq-notifications-server.yaml
2026-03-14 10:05:41.339375 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/rabbitmq-server.yaml
2026-03-14 10:05:41.339393 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/subscriptions/
2026-03-14 10:05:41.339410 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/
2026-03-14 10:05:41.339426 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/l2advertisement.yaml
2026-03-14 10:05:41.339443 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/ipaddresspools/
2026-03-14 10:05:41.339460 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/ipaddresspools/ctlplane.yaml
2026-03-14 10:05:41.339480 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/ipaddresspools/designateext.yaml
2026-03-14 10:05:41.339497 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/ipaddresspools/internalapi.yaml
2026-03-14 10:05:41.339514 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/ipaddresspools/storage.yaml
2026-03-14 10:05:41.339530 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/ipaddresspools/tenant.yaml
2026-03-14 10:05:41.339547 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/nnce/
2026-03-14 10:05:41.339564 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/nnce/crc.ens7-crc.yaml
2026-03-14 10:05:41.339580 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/nncp/
2026-03-14 10:05:41.339597 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/nncp/ens7-crc.yaml
2026-03-14 10:05:41.339614 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/nns/
2026-03-14 10:05:41.339631 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/nns/crc.yaml
2026-03-14 10:05:41.339649 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/nodes/
2026-03-14 10:05:41.339667 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/nodes/crc.describe
2026-03-14 10:05:41.339703 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/nodes/crc.yaml
2026-03-14 10:05:41.339722 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/sos-reports/
2026-03-14 10:05:41.339739 | controller | cL+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/sos-reports/sos-report- -> ../_all_nodes/sosreport-
2026-03-14 10:05:41.339757 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/sos-reports/_all_nodes/
2026-03-14 10:05:41.339779 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/
2026-03-14 10:05:41.339804 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/
2026-03-14 10:05:41.339828 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/cert-manager-webhook.yaml
2026-03-14 10:05:41.339850 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/infra-operator-mutating-webhook-configuration.yaml
2026-03-14 10:05:41.339868 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/mwatcher-v1beta1.kb.io-swcht.yaml
2026-03-14 10:05:41.339886 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/mwatcherapi-v1beta1.kb.io-kk5lq.yaml
2026-03-14 10:05:41.339908 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/mwatcherapplier-v1beta1.kb.io-tsmgs.yaml
2026-03-14 10:05:41.339929 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/mwatcherdecisionengine-v1beta1.kb.io-r5vfj.yaml
2026-03-14 10:05:41.339950 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/nmstate.yaml
2026-03-14 10:05:41.339972 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/openstack-baremetal-operator-mutating-webhook-configuration.yaml
2026-03-14 10:05:41.339994 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/openstack-operator-mutating-webhook-configuration.yaml
2026-03-14 10:05:41.340016 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/
2026-03-14 10:05:41.340038 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/alertmanagerconfigs.monitoring.rhobs-wwndb.yaml
2026-03-14 10:05:41.340060 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/bfdprofilevalidationwebhook.metallb.io-2vdss.yaml
2026-03-14 10:05:41.340083 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/bgpadvertisementvalidationwebhook.metallb.io-sxlmd.yaml
2026-03-14 10:05:41.340110 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/bgppeersvalidationwebhook.metallb.io-5444t.yaml
2026-03-14 10:05:41.340133 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/cert-manager-webhook.yaml
2026-03-14 10:05:41.340155 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/communityvalidationwebhook.metallb.io-mth9k.yaml
2026-03-14 10:05:41.340177 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/controlplanemachineset.machine.openshift.io.yaml
2026-03-14 10:05:41.340199 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/frr-k8s-validating-webhook-configuration.yaml
2026-03-14 10:05:41.340220 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/infra-operator-validating-webhook-configuration.yaml
2026-03-14 10:05:41.340241 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/ipaddresspoolvalidationwebhook.metallb.io-qvdkd.yaml
2026-03-14 10:05:41.340263 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/l2advertisementvalidationwebhook.metallb.io-ms8p5.yaml
2026-03-14 10:05:41.340285 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/metallbvalidationwebhook.metallb.io-gfrxn.yaml
2026-03-14 10:05:41.340306 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/monitoringconfigmaps.openshift.io.yaml
2026-03-14 10:05:41.340328 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/multus.openshift.io.yaml
2026-03-14 10:05:41.340350 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/network-node-identity.openshift.io.yaml
2026-03-14 10:05:41.340372 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/openstack-baremetal-operator-validating-webhook-configuration.yaml
2026-03-14 10:05:41.340393 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/openstack-operator-validating-webhook-configuration.yaml
2026-03-14 10:05:41.340419 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/prometheusrules.monitoring.rhobs-cxjhv.yaml
2026-03-14 10:05:41.340441 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/vwatcher-v1beta1.kb.io-lwc2q.yaml
2026-03-14 10:05:41.340463 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/vwatcherapi-v1beta1.kb.io-qrf7j.yaml
2026-03-14 10:05:41.340485 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/vwatcherapplier-v1beta1.kb.io-wcr2l.yaml
2026-03-14 10:05:41.340507 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/vwatcherdecisionengine-v1beta1.kb.io-97bg9.yaml
2026-03-14 10:05:41.340529 | controller | cd+++++++++ registries.conf.d/
2026-03-14 10:05:41.340553 | controller | >f+++++++++ registries.conf.d/000-shortnames.conf
2026-03-14 10:05:41.340576 | controller | cd+++++++++ system-config/
2026-03-14 10:05:41.340598 | controller | cd+++++++++ system-config/libvirt/
2026-03-14 10:05:41.340620 | controller | >f+++++++++ system-config/libvirt/libvirt-admin.conf
2026-03-14 10:05:41.340642 | controller | >f+++++++++ system-config/libvirt/libvirt.conf
2026-03-14 10:05:41.713189 | controller | ok
2026-03-14 10:05:42.265511 | controller | ok
2026-03-14 10:05:42.367741 | 
2026-03-14 10:05:42.367871 | TASK [Return artifact to Zuul]
2026-03-14 10:05:42.404305 | controller | skipping: Conditional result was False
2026-03-14 10:05:42.567379 | 
2026-03-14 10:05:42.567531 | PLAY [Run ci/playbooks/collect-logs.yml on CRC host]
2026-03-14 10:05:42.589057 | 
2026-03-14 10:05:42.589155 | TASK [Get kubelet journalctl logs]
2026-03-14 10:05:43.632863 | crc | Output suppressed because no_log was given
2026-03-14 10:05:43.647994 | 
2026-03-14 10:05:43.648127 | TASK [Compress logs bigger than 2MB]
2026-03-14 10:05:43.996999 | crc | gzip: /var/home/core/zuul-output/logs/kubelet.log.gz already exists;	not overwritten
2026-03-14 10:05:44.221323 | crc | ERROR
2026-03-14 10:05:44.221599 | crc | {
2026-03-14 10:05:44.221661 | crc |   "delta": "0:00:00.008036",
2026-03-14 10:05:44.221751 | crc |   "end": "2026-03-14 10:05:43.997480",
2026-03-14 10:05:44.221793 | crc |   "msg": "non-zero return code",
2026-03-14 10:05:44.221830 | crc |   "rc": 1,
2026-03-14 10:05:44.221868 | crc |   "start": "2026-03-14 10:05:43.989444"
2026-03-14 10:05:44.221905 | crc | }
2026-03-14 10:05:44.221953 | crc | ERROR: Ignoring Errors
2026-03-14 10:05:44.239534 | 
2026-03-14 10:05:44.239713 | TASK [Copy files from workspace on node]
2026-03-14 10:05:44.272551 | crc | ok
2026-03-14 10:05:44.305202 | 
2026-03-14 10:05:44.305296 | TASK [fetch-output : Set log path for multiple nodes]
2026-03-14 10:05:44.355840 | crc | ok
2026-03-14 10:05:44.370310 | 
2026-03-14 10:05:44.370396 | TASK [fetch-output : Set log path for single node]
2026-03-14 10:05:44.395936 | crc | skipping: Conditional result was False
2026-03-14 10:05:44.407625 | 
2026-03-14 10:05:44.407741 | LOOP [fetch-output : Ensure local output dirs]
2026-03-14 10:05:44.686139 | crc -> localhost | ok: "/var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/logs/crc"
2026-03-14 10:05:44.686386 | crc -> localhost | changed: All items complete
2026-03-14 10:05:44.686413 | 
2026-03-14 10:05:44.892203 | crc -> localhost | ok: "/var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/artifacts"
2026-03-14 10:05:45.097596 | crc -> localhost | ok: "/var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/docs"
2026-03-14 10:05:45.125361 | 
2026-03-14 10:05:45.125565 | LOOP [fetch-output : Collect logs, artifacts and docs]
2026-03-14 10:05:45.848294 | crc | changed:
2026-03-14 10:05:45.848760 | crc | .d..t...... ./
2026-03-14 10:05:45.848828 | crc | >f+++++++++ kubelet.log
2026-03-14 10:05:46.359096 | crc | ok
2026-03-14 10:05:46.862643 | crc | ok
2026-03-14 10:05:46.932383 | 
2026-03-14 10:05:46.932509 | PLAY RECAP
2026-03-14 10:05:46.932572 | controller | ok: 15 changed: 11 unreachable: 0 failed: 0 skipped: 7 rescued: 0 ignored: 1
2026-03-14 10:05:46.932609 | crc | ok: 5 changed: 3 unreachable: 0 failed: 0 skipped: 1 rescued: 0 ignored: 1
2026-03-14 10:05:46.932635 | 
2026-03-14 10:05:47.056846 | POST-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml@main]
2026-03-14 10:05:47.062880 | POST-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/multinode-autohold.yml@main]
2026-03-14 10:05:47.607426 | 
2026-03-14 10:05:47.607578 | PLAY [Run ci/playbooks/multinode-autohold.yml]
2026-03-14 10:05:47.686010 | 
2026-03-14 10:05:47.686154 | TASK [Verify if "success" flag exists after successful tests execution]
2026-03-14 10:05:48.290309 | controller | ok
2026-03-14 10:05:48.314750 | 
2026-03-14 10:05:48.314841 | TASK [Fetch existing autoholds from zuul]
2026-03-14 10:05:48.394248 | controller | ok
2026-03-14 10:05:48.443608 | 
2026-03-14 10:05:48.443736 | TASK [krb_request : Perform the request]
2026-03-14 10:05:49.111857 | controller | ok
2026-03-14 10:05:49.197941 | 
2026-03-14 10:05:49.198070 | TASK [Check if any autohold matches]
2026-03-14 10:05:49.293733 | controller | skipping: Conditional result was False
2026-03-14 10:05:49.386510 | 
2026-03-14 10:05:49.386614 | PLAY RECAP
2026-03-14 10:05:49.386656 | controller | ok: 2 changed: 0 unreachable: 0 failed: 0 skipped: 1 rescued: 0 ignored: 0
2026-03-14 10:05:49.386700 | 
2026-03-14 10:05:49.487864 | POST-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/multinode-autohold.yml@main]
2026-03-14 10:05:49.492499 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/print-machineconfigpool.yaml@master]
2026-03-14 10:05:50.092279 | 
2026-03-14 10:05:50.092437 | PLAY [crc]
2026-03-14 10:05:50.119003 | 
2026-03-14 10:05:50.119092 | TASK [Print machineconfigpool]
2026-03-14 10:05:51.039491 | crc | NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
2026-03-14 10:05:51.046171 | crc | master   rendered-master-f83a149c7490c3b0bf850df839837383   False     False      True       1              0                   0                     1                      384d
2026-03-14 10:05:51.046251 | crc | worker   rendered-worker-0dfa5a008ca491b87cb16c8a318e0cef   True      False      False      0              0                   0                     0                      384d
2026-03-14 10:05:51.157765 | crc | changed
2026-03-14 10:05:51.210416 | 
2026-03-14 10:05:51.210502 | PLAY RECAP
2026-03-14 10:05:51.210559 | crc | ok: 1 changed: 1 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0
2026-03-14 10:05:51.210587 | 
2026-03-14 10:05:51.306349 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/print-machineconfigpool.yaml@master]
2026-03-14 10:05:51.317927 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/compress-crc-cloud-logs.yaml@master]
2026-03-14 10:05:51.884424 | 
2026-03-14 10:05:51.884577 | PLAY [crc]
2026-03-14 10:05:51.910951 | 
2026-03-14 10:05:51.911058 | TASK [Create compressed log tar gz file - crc-cloud logs]
2026-03-14 10:05:52.599997 | crc | tar: Removing leading `/' from member names
2026-03-14 10:05:52.601684 | crc | tar: /var/home/core/workdir/*log: Cannot stat: No such file or directory
2026-03-14 10:05:52.601733 | crc | tar: Exiting with failure status due to previous errors
2026-03-14 10:05:52.948881 | crc | ok
2026-03-14 10:05:52.964558 | 
2026-03-14 10:05:52.964756 | TASK [Copy crc-cloud logs]
2026-03-14 10:05:53.380445 | crc | cp: cannot stat '/var/home/core/workdir/*log': No such file or directory
2026-03-14 10:05:53.532137 | crc | ok
2026-03-14 10:05:53.557826 | 
2026-03-14 10:05:53.557931 | PLAY RECAP
2026-03-14 10:05:53.557979 | crc | ok: 2 changed: 0 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0
2026-03-14 10:05:53.558005 | 
2026-03-14 10:05:53.658375 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/compress-crc-cloud-logs.yaml@master]
2026-03-14 10:05:53.668309 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/compress-logs.yaml@master]
2026-03-14 10:05:54.236999 | 
2026-03-14 10:05:54.237143 | PLAY [all]
2026-03-14 10:05:54.325785 | 
2026-03-14 10:05:54.325907 | TASK [Create compressed log tar gz file - ci-framework-data]
2026-03-14 10:05:54.979470 | controller | tar: Removing leading `/' from member names
2026-03-14 10:05:55.252176 | crc | tar: Removing leading `/' from member names
2026-03-14 10:05:55.254479 | crc | tar: /var/home/core/ci-framework-data: Cannot stat: No such file or directory
2026-03-14 10:05:55.254506 | crc | tar: Exiting with failure status due to previous errors
2026-03-14 10:05:55.257208 | crc | mv: cannot move '/var/home/core/ci-framework-data-crc-all-logs.tar.gz' to '/var/home/core/ci-framework-data/logs/ci-framework-data-crc-all-logs.tar.gz': No such file or directory
2026-03-14 10:05:55.260499 | crc | chown: cannot access '/var/home/core/ci-framework-data/logs/ci-framework-data-crc-all-logs.tar.gz': No such file or directory
2026-03-14 10:05:55.510323 | crc | ERROR
2026-03-14 10:05:55.510564 | crc | {
2026-03-14 10:05:55.510597 | crc |   "delta": "0:00:00.014742",
2026-03-14 10:05:55.510618 | crc |   "end": "2026-03-14 10:05:55.260781",
2026-03-14 10:05:55.510635 | crc |   "msg": "non-zero return code",
2026-03-14 10:05:55.510651 | crc |   "rc": 1,
2026-03-14 10:05:55.510668 | crc |   "start": "2026-03-14 10:05:55.246039"
2026-03-14 10:05:55.510704 | crc | }
2026-03-14 10:05:55.510737 | crc | ERROR: Ignoring Errors
2026-03-14 10:06:03.465957 | controller | ok
2026-03-14 10:06:03.498957 | 
2026-03-14 10:06:03.499089 | TASK [Create compressed log tar gz file - zuul-output]
2026-03-14 10:06:03.965664 | controller | tar: Removing leading `/' from member names
2026-03-14 10:06:04.019563 | crc | tar: Removing leading `/' from member names
2026-03-14 10:06:04.219813 | crc | ok
2026-03-14 10:06:06.676847 | controller | ok
2026-03-14 10:06:06.837165 | 
2026-03-14 10:06:06.837273 | PLAY RECAP
2026-03-14 10:06:06.837312 | controller | ok: 2 changed: 0 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0
2026-03-14 10:06:06.837347 | crc | ok: 2 changed: 0 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 1
2026-03-14 10:06:06.837373 | 
2026-03-14 10:06:06.951407 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/compress-logs.yaml@master]
2026-03-14 10:06:06.961485 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/bootstraped-ci-network-cleanup.yml@master]
2026-03-14 10:06:07.566156 | 
2026-03-14 10:06:07.566271 | PLAY [Run playbooks/crc/bootstraped-ci-network-cleanup.yml]
2026-03-14 10:06:07.641803 | 
2026-03-14 10:06:07.641930 | TASK [Create openstack config dir]
2026-03-14 10:06:08.280404 | controller | ok
2026-03-14 10:06:08.305460 | 
2026-03-14 10:06:08.305538 | TASK [Check for cleanup skip flag file presence]
2026-03-14 10:06:08.737643 | controller | ok
2026-03-14 10:06:08.769939 | 
2026-03-14 10:06:08.770033 | TASK [Generate clouds config from cloud_secrets secret]
2026-03-14 10:06:09.725270 | controller | changed
2026-03-14 10:06:09.756833 | 
2026-03-14 10:06:09.756909 | LOOP [Delete ports from each host]
2026-03-14 10:06:09.930752 | controller | ok: "crc"
2026-03-14 10:06:12.883901 | controller | ok: "controller"
2026-03-14 10:06:12.951067 | controller | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstraped-ci-network-cleanup-instance-ports.yml
2026-03-14 10:06:12.955777 | controller | included: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstraped-ci-network-cleanup-instance-ports.yml
2026-03-14 10:06:12.989147 | 
2026-03-14 10:06:12.989244 | TASK [Fetch port info]
2026-03-14 10:06:15.263950 | controller | admin_state_up: true
2026-03-14 10:06:15.264291 | controller | allowed_address_pairs: []
2026-03-14 10:06:15.264709 | controller | binding_host_id: null
2026-03-14 10:06:15.264948 | controller | binding_profile: {}
2026-03-14 10:06:15.265176 | controller | binding_vif_details: {}
2026-03-14 10:06:15.265413 | controller | binding_vif_type: null
2026-03-14 10:06:15.265735 | controller | binding_vnic_type: normal
2026-03-14 10:06:15.266005 | controller | created_at: '2026-03-14T09:08:42Z'
2026-03-14 10:06:15.266233 | controller | data_plane_status: null
2026-03-14 10:06:15.266452 | controller | description: ''
2026-03-14 10:06:15.266748 | controller | device_id: 7ad45a7c-2b2c-4e9f-b1c1-584bf6854f24
2026-03-14 10:06:15.266983 | controller | device_owner: compute:nova
2026-03-14 10:06:15.267206 | controller | device_profile: null
2026-03-14 10:06:15.267925 | controller | dns_assignment:
2026-03-14 10:06:15.268142 | controller | - fqdn: host-192-168-122-10.openstacklocal.
2026-03-14 10:06:15.268152 | controller |   hostname: host-192-168-122-10
2026-03-14 10:06:15.268160 | controller |   ip_address: 192.168.122.10
2026-03-14 10:06:15.268167 | controller | dns_domain: ''
2026-03-14 10:06:15.268351 | controller | dns_name: ''
2026-03-14 10:06:15.268617 | controller | extra_dhcp_opts: []
2026-03-14 10:06:15.269200 | controller | fixed_ips:
2026-03-14 10:06:15.269447 | controller | - ip_address: 192.168.122.10
2026-03-14 10:06:15.269458 | controller |   subnet_id: 3293acd3-87a9-4c81-8422-4b391ba88681
2026-03-14 10:06:15.269465 | controller | hardware_offload_type: null
2026-03-14 10:06:15.269666 | controller | hints: ''
2026-03-14 10:06:15.269929 | controller | id: 41aa7b96-5e47-4b38-affe-d761cc588e1f
2026-03-14 10:06:15.270153 | controller | ip_allocation: immediate
2026-03-14 10:06:15.270393 | controller | mac_address: fa:16:3e:27:7a:64
2026-03-14 10:06:15.270696 | controller | name: crc-7ad45a7c-2b2c-4e9f-b1c1-584bf6854f24
2026-03-14 10:06:15.270976 | controller | network_id: 993bd7ea-a6c0-4049-a480-d39fe80f9b55
2026-03-14 10:06:15.271210 | controller | numa_affinity_policy: null
2026-03-14 10:06:15.271510 | controller | port_security_enabled: false
2026-03-14 10:06:15.271805 | controller | project_id: 4b633c451ac74233be3721a3635275e5
2026-03-14 10:06:15.272042 | controller | propagate_uplink_status: null
2026-03-14 10:06:15.272266 | controller | resource_request: null
2026-03-14 10:06:15.272579 | controller | revision_number: 4
2026-03-14 10:06:15.272843 | controller | qos_network_policy_id: null
2026-03-14 10:06:15.273111 | controller | qos_policy_id: null
2026-03-14 10:06:15.273363 | controller | security_group_ids: []
2026-03-14 10:06:15.273570 | controller | status: ACTIVE
2026-03-14 10:06:15.273812 | controller | tags: []
2026-03-14 10:06:15.274039 | controller | trunk_details: null
2026-03-14 10:06:15.274246 | controller | trusted: null
2026-03-14 10:06:15.274501 | controller | updated_at: '2026-03-14T09:08:56Z'
2026-03-14 10:06:15.640793 | controller | changed
2026-03-14 10:06:15.673735 | 
2026-03-14 10:06:15.673841 | TASK [Set port info yaml for further usage]
2026-03-14 10:06:15.793990 | controller | ok
2026-03-14 10:06:15.853125 | 
2026-03-14 10:06:15.853322 | TASK [Detach port]
2026-03-14 10:06:19.512102 | controller | changed
2026-03-14 10:06:19.535225 | 
2026-03-14 10:06:19.535333 | LOOP [Delete port association with trunk network]
2026-03-14 10:06:19.625640 | 
2026-03-14 10:06:19.625798 | LOOP [Delete trunk sub ports (tagged ones)]
2026-03-14 10:06:19.713423 | 
2026-03-14 10:06:19.713639 | TASK [Delete the trunk network itself]
2026-03-14 10:06:20.374038 | controller | skipping: Conditional result was False
2026-03-14 10:06:20.427421 | 
2026-03-14 10:06:20.427535 | TASK [Remove the parent/trunk port itself]
2026-03-14 10:06:33.106961 | controller | changed
2026-03-14 10:06:33.143207 | 
2026-03-14 10:06:33.143355 | TASK [Fetch port info]
2026-03-14 10:06:41.443279 | controller | admin_state_up: true
2026-03-14 10:06:41.443663 | controller | allowed_address_pairs: []
2026-03-14 10:06:41.444046 | controller | binding_host_id: null
2026-03-14 10:06:41.444414 | controller | binding_profile: {}
2026-03-14 10:06:41.444688 | controller | binding_vif_details: {}
2026-03-14 10:06:41.444920 | controller | binding_vif_type: null
2026-03-14 10:06:41.445197 | controller | binding_vnic_type: normal
2026-03-14 10:06:41.445443 | controller | created_at: '2026-03-14T09:08:28Z'
2026-03-14 10:06:41.445680 | controller | data_plane_status: null
2026-03-14 10:06:41.445877 | controller | description: ''
2026-03-14 10:06:41.446121 | controller | device_id: f396667f-02e9-4d75-b54e-bb319e7c18b8
2026-03-14 10:06:41.446331 | controller | device_owner: compute:nova
2026-03-14 10:06:41.446533 | controller | device_profile: null
2026-03-14 10:06:41.447190 | controller | dns_assignment:
2026-03-14 10:06:41.447388 | controller | - fqdn: host-192-168-122-11.openstacklocal.
2026-03-14 10:06:41.447394 | controller |   hostname: host-192-168-122-11
2026-03-14 10:06:41.447399 | controller |   ip_address: 192.168.122.11
2026-03-14 10:06:41.447406 | controller | dns_domain: ''
2026-03-14 10:06:41.447574 | controller | dns_name: ''
2026-03-14 10:06:41.447809 | controller | extra_dhcp_opts: []
2026-03-14 10:06:41.448345 | controller | fixed_ips:
2026-03-14 10:06:41.448566 | controller | - ip_address: 192.168.122.11
2026-03-14 10:06:41.448573 | controller |   subnet_id: 3293acd3-87a9-4c81-8422-4b391ba88681
2026-03-14 10:06:41.448579 | controller | hardware_offload_type: null
2026-03-14 10:06:41.448773 | controller | hints: ''
2026-03-14 10:06:41.449029 | controller | id: 7613f3f5-a135-4006-852d-a68a6e849070
2026-03-14 10:06:41.449239 | controller | ip_allocation: immediate
2026-03-14 10:06:41.449456 | controller | mac_address: fa:16:3e:23:c7:77
2026-03-14 10:06:41.449735 | controller | name: controller-f396667f-02e9-4d75-b54e-bb319e7c18b8
2026-03-14 10:06:41.449999 | controller | network_id: 993bd7ea-a6c0-4049-a480-d39fe80f9b55
2026-03-14 10:06:41.450222 | controller | numa_affinity_policy: null
2026-03-14 10:06:41.450501 | controller | port_security_enabled: false
2026-03-14 10:06:41.450777 | controller | project_id: 4b633c451ac74233be3721a3635275e5
2026-03-14 10:06:41.451008 | controller | propagate_uplink_status: null
2026-03-14 10:06:41.451223 | controller | resource_request: null
2026-03-14 10:06:41.451491 | controller | revision_number: 4
2026-03-14 10:06:41.452005 | controller | qos_network_policy_id: null
2026-03-14 10:06:41.452321 | controller | qos_policy_id: null
2026-03-14 10:06:41.452543 | controller | security_group_ids: []
2026-03-14 10:06:41.452788 | controller | status: ACTIVE
2026-03-14 10:06:41.452986 | controller | tags: []
2026-03-14 10:06:41.453188 | controller | trunk_details: null
2026-03-14 10:06:41.453380 | controller | trusted: null
2026-03-14 10:06:41.453639 | controller | updated_at: '2026-03-14T09:08:40Z'
2026-03-14 10:06:41.830387 | controller | changed
2026-03-14 10:06:41.882083 | 
2026-03-14 10:06:41.882183 | TASK [Set port info yaml for further usage]
2026-03-14 10:06:42.047910 | controller | ok
2026-03-14 10:06:42.070376 | 
2026-03-14 10:06:42.070443 | TASK [Detach port]
2026-03-14 10:06:54.730370 | controller | changed
2026-03-14 10:06:54.781401 | 
2026-03-14 10:06:54.781581 | LOOP [Delete port association with trunk network]
2026-03-14 10:06:54.897556 | 
2026-03-14 10:06:54.897761 | LOOP [Delete trunk sub ports (tagged ones)]
2026-03-14 10:06:54.989916 | 
2026-03-14 10:06:54.990123 | TASK [Delete the trunk network itself]
2026-03-14 10:06:55.649253 | controller | skipping: Conditional result was False
2026-03-14 10:06:55.704887 | 
2026-03-14 10:06:55.705033 | TASK [Remove the parent/trunk port itself]
2026-03-14 10:07:09.391665 | controller | changed
2026-03-14 10:07:09.415811 | 
2026-03-14 10:07:09.415915 | TASK [Dettach router from subnet]
2026-03-14 10:07:10.036188 | controller | skipping: Conditional result was False
2026-03-14 10:07:10.130924 | 
2026-03-14 10:07:10.131041 | TASK [Delete router]
2026-03-14 10:07:10.754432 | controller | skipping: Conditional result was False
2026-03-14 10:07:10.778430 | 
2026-03-14 10:07:10.778502 | TASK [Delete subnet]
2026-03-14 10:07:22.988859 | controller | changed
2026-03-14 10:07:23.026731 | 
2026-03-14 10:07:23.026830 | TASK [Delete network]
2026-03-14 10:07:36.199819 | controller | changed
2026-03-14 10:07:36.249972 | 
2026-03-14 10:07:36.250105 | TASK [Remove clouds.yml secret]
2026-03-14 10:07:36.554386 | controller | changed
2026-03-14 10:07:36.691879 | 
2026-03-14 10:07:36.691982 | PLAY RECAP
2026-03-14 10:07:36.692022 | controller | ok: 16 changed: 10 unreachable: 0 failed: 0 skipped: 8 rescued: 0 ignored: 0
2026-03-14 10:07:36.692042 | 
2026-03-14 10:07:36.793058 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/bootstraped-ci-network-cleanup.yml@master]
2026-03-14 10:07:36.802953 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/ci-framework-rdo-base/dlrn/dlrn-report.yaml@master]
2026-03-14 10:07:37.425117 | 
2026-03-14 10:07:37.425228 | PLAY [Report DLRN results after job run]
2026-03-14 10:07:37.501975 | 
2026-03-14 10:07:37.502103 | TASK [Report job status to DLRN]
2026-03-14 10:07:37.566678 | controller | ok
2026-03-14 10:07:37.625656 | 
2026-03-14 10:07:37.625790 | TASK [dlrn_report : Warning if DLRN is not running]
2026-03-14 10:07:37.701428 | controller | skipping: Conditional result was False
2026-03-14 10:07:37.747872 | 
2026-03-14 10:07:37.747990 | TASK [dlrn_report : Install dlrnapi-client shyaml package]
2026-03-14 10:07:37.825176 | controller | skipping: Conditional result was False
2026-03-14 10:07:37.875877 | 
2026-03-14 10:07:37.876009 | TASK [dlrn_report : Install kinit related package]
2026-03-14 10:07:37.953265 | controller | skipping: Conditional result was False
2026-03-14 10:07:38.005075 | 
2026-03-14 10:07:38.005199 | TASK [dlrn_report : Install dlrn kerberos related packages]
2026-03-14 10:07:38.082319 | controller | skipping: Conditional result was False
2026-03-14 10:07:38.174163 | 
2026-03-14 10:07:38.174283 | TASK [Set zuul-log-path fact]
2026-03-14 10:07:38.250323 | controller | skipping: Conditional result was False
2026-03-14 10:07:38.301503 | 
2026-03-14 10:07:38.301634 | TASK [dlrn_report : Set the the value of cifmw_repo_setup_promotion when multiple dlrn tags are used]
2026-03-14 10:07:38.379251 | controller | skipping: Conditional result was False
2026-03-14 10:07:38.431117 | 
2026-03-14 10:07:38.431254 | TASK [Get hash related data from repo_setup role]
2026-03-14 10:07:38.507879 | controller | skipping: Conditional result was False
2026-03-14 10:07:38.559428 | 
2026-03-14 10:07:38.559583 | TASK [dlrn_report : Perform kinit for DLRN kerberos authentication]
2026-03-14 10:07:39.224634 | controller | skipping: Conditional result was False
2026-03-14 10:07:39.277809 | 
2026-03-14 10:07:39.277940 | TASK [dlrn_report : Set empty value for dlrnapi password]
2026-03-14 10:07:39.323241 | controller | skipping: Conditional result was False
2026-03-14 10:07:39.372421 | 
2026-03-14 10:07:39.372546 | TASK [dlrn_report : Report results to dlrn for the tested hash]
2026-03-14 10:07:40.029190 | controller | skipping: Conditional result was False
2026-03-14 10:07:40.213301 | 
2026-03-14 10:07:40.213412 | PLAY RECAP
2026-03-14 10:07:40.213451 | controller | ok: 0 changed: 0 unreachable: 0 failed: 0 skipped: 10 rescued: 0 ignored: 0
2026-03-14 10:07:40.213472 | 
2026-03-14 10:07:40.303909 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/ci-framework-rdo-base/dlrn/dlrn-report.yaml@master]
2026-03-14 10:07:40.314433 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-ssh.yaml@master]
2026-03-14 10:07:40.947912 | 
2026-03-14 10:07:40.948021 | PLAY [all]
2026-03-14 10:07:41.063460 | 
2026-03-14 10:07:41.063602 | TASK [include_role : fetch-output]
2026-03-14 10:07:41.126016 | controller | ok
2026-03-14 10:07:41.133011 | crc | ok
2026-03-14 10:07:41.193799 | 
2026-03-14 10:07:41.193917 | TASK [fetch-output : Set log path for multiple nodes]
2026-03-14 10:07:41.299297 | controller | ok
2026-03-14 10:07:41.305781 | crc | ok
2026-03-14 10:07:41.328256 | 
2026-03-14 10:07:41.328324 | TASK [fetch-output : Set log path for single node]
2026-03-14 10:07:41.365227 | controller | skipping: Conditional result was False
2026-03-14 10:07:41.386125 | crc | skipping: Conditional result was False
2026-03-14 10:07:41.448359 | 
2026-03-14 10:07:41.448488 | LOOP [fetch-output : Ensure local output dirs]
2026-03-14 10:07:41.806501 | crc -> localhost | ok: "/var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/logs/crc"
2026-03-14 10:07:41.815033 | controller -> localhost | ok: "/var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/logs/controller"
2026-03-14 10:07:42.022686 | crc -> localhost | ok: "/var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/artifacts"
2026-03-14 10:07:42.045782 | controller -> localhost | ok: "/var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/artifacts"
2026-03-14 10:07:42.220030 | crc -> localhost | ok: "/var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/docs"
2026-03-14 10:07:42.241442 | controller -> localhost | ok: "/var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/docs"
2026-03-14 10:07:42.241577 | controller -> localhost | ok: All items complete
2026-03-14 10:07:42.241603 | 
2026-03-14 10:07:42.277712 | 
2026-03-14 10:07:42.277845 | LOOP [fetch-output : Collect logs, artifacts and docs]
2026-03-14 10:07:43.070892 | crc | changed:
2026-03-14 10:07:43.071486 | crc | .d..t...... ./
2026-03-14 10:07:43.071560 | crc | >f+++++++++ crc-cloud-workdir-crc-all-logs.tar.gz
2026-03-14 10:07:43.071608 | crc | >f+++++++++ zuul-output-crc-all-logs.tar.gz
2026-03-14 10:07:43.071653 | crc | cd+++++++++ crc-cloud/
2026-03-14 10:07:43.648975 | crc | ok
2026-03-14 10:07:44.013479 | controller | changed:
2026-03-14 10:07:44.013633 | controller | .d..t...... ./
2026-03-14 10:07:44.013672 | controller | >f+++++++++ zuul-output-controller-all-logs.tar.gz
2026-03-14 10:07:44.169564 | crc | ok
2026-03-14 10:07:44.498046 | controller | ok
2026-03-14 10:07:44.498190 | controller | changed: All items complete
2026-03-14 10:07:44.498235 | 
2026-03-14 10:07:45.006574 | controller | ok
2026-03-14 10:07:45.128500 | 
2026-03-14 10:07:45.128629 | TASK [include_role : fetch-output-openshift]
2026-03-14 10:07:45.155593 | controller | skipping: Conditional result was False
2026-03-14 10:07:45.166437 | crc | skipping: Conditional result was False
2026-03-14 10:07:45.191084 | 
2026-03-14 10:07:45.191153 | LOOP [merge-output-to-logs : Move artifacts and docs to logs dir]
2026-03-14 10:07:45.697960 | controller -> localhost | ok: Item: artifacts Runtime: 0:00:00.012935
2026-03-14 10:07:45.986590 | controller -> localhost | ok: Item: docs Runtime: 0:00:00.013778
2026-03-14 10:07:46.176547 | 
2026-03-14 10:07:46.176652 | PLAY [all]
2026-03-14 10:07:46.256786 | 
2026-03-14 10:07:46.256903 | TASK [remove-build-sshkey : Remove the build SSH key from all nodes]
2026-03-14 10:07:46.963006 | controller | changed
2026-03-14 10:07:47.195662 | crc | changed
2026-03-14 10:07:47.389460 | 
2026-03-14 10:07:47.389564 | PLAY RECAP
2026-03-14 10:07:47.389605 | controller | ok: 5 changed: 3 unreachable: 0 failed: 0 skipped: 2 rescued: 0 ignored: 0
2026-03-14 10:07:47.389633 | crc | ok: 4 changed: 2 unreachable: 0 failed: 0 skipped: 2 rescued: 0 ignored: 0
2026-03-14 10:07:47.389651 | 
2026-03-14 10:07:47.487320 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-ssh.yaml@master]
2026-03-14 10:07:47.491827 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-logs.yaml@master]
2026-03-14 10:07:48.130849 | 
2026-03-14 10:07:48.130957 | PLAY [localhost]
2026-03-14 10:07:48.149349 | 
2026-03-14 10:07:48.149437 | TASK [generate-zuul-manifest : Generate Zuul manifest]
2026-03-14 10:07:48.789331 | localhost | changed
2026-03-14 10:07:48.795764 | 
2026-03-14 10:07:48.795872 | TASK [generate-zuul-manifest : Return Zuul manifest URL to Zuul]
2026-03-14 10:07:48.830118 | localhost | ok
2026-03-14 10:07:48.847792 | 
2026-03-14 10:07:48.847933 | TASK [add-fileserver : Create SSH private key tempfile]
2026-03-14 10:07:49.250207 | localhost | changed
2026-03-14 10:07:49.260661 | 
2026-03-14 10:07:49.260902 | TASK [add-fileserver : Create SSH private key from secret]
2026-03-14 10:07:50.055552 | localhost | changed
2026-03-14 10:07:50.068594 | 
2026-03-14 10:07:50.068771 | TASK [add-fileserver : Add fileserver ssh key]
2026-03-14 10:07:50.495283 | localhost | Identity added: /var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/tmp/ansible.lq9bpz0h (/var/lib/zuul/builds/bd4c83535f9d4245993d8306726273fc/work/tmp/ansible.lq9bpz0h)
2026-03-14 10:07:50.495589 | localhost | ok: Runtime: 0:00:00.006892
2026-03-14 10:07:50.505477 | 
2026-03-14 10:07:50.505618 | TASK [add-fileserver : Remove SSH private key from disk]
2026-03-14 10:07:50.821855 | localhost | ok: Runtime: 0:00:00.008888
2026-03-14 10:07:50.846960 | 
2026-03-14 10:07:50.847075 | TASK [add-fileserver : Add fileserver to inventory]
2026-03-14 10:07:50.926190 | localhost | changed
2026-03-14 10:07:50.930461 | 
2026-03-14 10:07:50.930524 | TASK [add-fileserver : Add fileserver server to known hosts]
2026-03-14 10:07:51.302541 | localhost | changed
2026-03-14 10:07:51.322650 | 
2026-03-14 10:07:51.322783 | PLAY [localhost]
2026-03-14 10:07:51.335952 | 
2026-03-14 10:07:51.336011 | TASK [Generate bulk log download script]
2026-03-14 10:07:51.354376 | localhost | ok
2026-03-14 10:07:51.366573 | 
2026-03-14 10:07:51.366636 | TASK [local-log-download : Check API endpoint is defined]
2026-03-14 10:07:51.405980 | localhost | ok: All assertions passed
2026-03-14 10:07:51.410411 | 
2026-03-14 10:07:51.410469 | TASK [local-log-download : Create download script]
2026-03-14 10:07:51.841228 | localhost -> localhost | changed
2026-03-14 10:07:51.849916 | 
2026-03-14 10:07:51.849987 | TASK [Register quick-download link]
2026-03-14 10:07:51.868753 | localhost | ok
2026-03-14 10:07:51.883867 | 
2026-03-14 10:07:51.883918 | PLAY [logserver.rdoproject.org]
2026-03-14 10:07:51.893416 | 
2026-03-14 10:07:51.893475 | TASK [Set zuul-log-path fact]
2026-03-14 10:07:51.910470 | logserver.rdoproject.org | ok
2026-03-14 10:07:51.920069 | 
2026-03-14 10:07:51.920138 | TASK [set-zuul-log-path-fact : Set log path for a build]
2026-03-14 10:07:51.957057 | logserver.rdoproject.org | ok
2026-03-14 10:07:51.962335 | 
2026-03-14 10:07:51.962394 | TASK [upload-logs : Create log directories]
2026-03-14 10:07:52.929080 | logserver.rdoproject.org | changed
2026-03-14 10:07:52.935783 | 
2026-03-14 10:07:52.935913 | TASK [upload-logs : Ensure logs are readable before uploading]
2026-03-14 10:07:53.328109 | logserver.rdoproject.org -> localhost | ok: Runtime: 0:00:00.055949
2026-03-14 10:07:53.337603 | 
2026-03-14 10:07:53.337774 | TASK [upload-logs : Upload logs to log server]
2026-03-14 10:08:00.003375 | logserver.rdoproject.org | Output suppressed because no_log was given
2026-03-14 10:08:00.009596 | 
2026-03-14 10:08:00.009803 | LOOP [upload-logs : Compress console log and json output]
2026-03-14 10:08:00.064867 | logserver.rdoproject.org | skipping: Conditional result was False
2026-03-14 10:08:00.077353 | logserver.rdoproject.org | skipping: Conditional result was False
2026-03-14 10:08:00.095932 | 
2026-03-14 10:08:00.096078 | LOOP [upload-logs : Upload compressed console log and json output]
2026-03-14 10:08:00.146501 | logserver.rdoproject.org | skipping: Conditional result was False
2026-03-14 10:08:00.146984 | 
2026-03-14 10:08:00.149670 | logserver.rdoproject.org | skipping: Conditional result was False
2026-03-14 10:08:00.161135 | 
2026-03-14 10:08:00.161280 | LOOP [upload-logs : Upload console log and json output]